deploy to HF Space

.gitattributes

@@ -0,0 +1,2 @@
+*.jsonl filter=lfs diff=lfs merge=lfs -text
+*.pdf filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,69 @@
+.idea/COLE.iml
+.idea/inspectionProfiles/profiles_settings.xml
+.idea/misc.xml
+.idea/modules.xml
+.idea/vcs.xml
+.idea/workspace.xml
+.idea/*
+/Benchmarks
+/__pycache__
+src/config.py
+/Loaded_Models
+/results
+/hf_data
+
+
+COLE_nlu_benchmark_site/node_modules
+COLE_nlu_benchmark_site/.pnp
+COLE_nlu_benchmark_site/.pnp.*
+COLE_nlu_benchmark_site/.yarn/*
+!.yarn/patches
+!.yarn/plugins
+!.yarn/releases
+!.yarn/versions
+
+# testing
+/coverage
+.coverage
+coverage.xml
+htmlcov/
+
+# next.js
+COLE_nlu_benchmark_site/.next/
+COLE_nlu_benchmark_site/out/
+
+# production
+COLE_nlu_benchmark_site/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+
+# env files (can opt-in for committing if needed)
+.env*
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts
+/src/results
+/src/__pycache__
+/src/backend/__pycache__
+/archives/Models/__pycache__
+*.pyc
+/Benchmarks_data
+/src/light_eval_custom/results
+/offline_evaluation/results
+/frontend/node_modules
+/.idea/*
+/frontend/.next
+/src/backend/results
+/node_modules

.idea/.gitignore

@@ -0,0 +1,8 @@
+# Default ignored files
+/shelf/
+/workspace.xml
+# Editor-based HTTP Client requests
+/httpRequests/
+# Datasource local storage ignored files
+/dataSources/
+/dataSources.local.xml

.pre-commit-config.yaml

@@ -0,0 +1,27 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 26.3.1
+    hooks:
+      - id: black
+
+  - repo: https://github.com/PyCQA/pylint
+    rev: v3.3.6
+    hooks:
+      - id: pylint
+        args: [cole/, tests/]
+        pass_filenames: false
+        additional_dependencies:
+          - fastapi
+          - uvicorn
+          - slowapi
+          - httpx
+          - pydantic
+
+  - repo: local
+    hooks:
+      - id: eslint
+        name: eslint
+        entry: bash -c 'cd frontend && npm run lint'
+        language: system
+        files: ^frontend/.*\.(js|jsx|ts|tsx)$
+        pass_filenames: false

CITATION.cff

@@ -0,0 +1,40 @@
+cff-version: 1.2.0
+title: "COLE: a Comprehensive Benchmark for French Language Understanding Evaluation"
+message: "If you use COLE in your research, please cite our paper."
+type: software
+authors:
+  - given-names: David
+    family-names: Beauchemin
+    affiliation: Université Laval
+  - given-names: Yan
+    family-names: Tremblay
+    affiliation: Université Laval
+  - given-names: Mohamed Amine
+    family-names: Youssef
+    affiliation: Université Laval
+  - given-names: Richard
+    family-names: Khoury
+    affiliation: Université Laval
+repository-code: "https://github.com/GRAAL-Research/COLE"
+url: "https://colebenchmark.org"
+license: MIT
+version: 1.0.0
+date-released: "2025-10-07"
+preferred-citation:
+  type: article
+  title: "COLE: a Comprehensive Benchmark for French Language Understanding Evaluation"
+  authors:
+    - given-names: David
+      family-names: Beauchemin
+    - given-names: Yan
+      family-names: Tremblay
+    - given-names: Mohamed Amine
+      family-names: Youssef
+    - given-names: Richard
+      family-names: Khoury
+  year: 2025
+  url: "https://arxiv.org/abs/2510.05046"
+  identifiers:
+    - type: other
+      value: "arXiv:2510.05046"
+      description: arXiv preprint

CLAUDE.md

@@ -0,0 +1,70 @@
+# COLE — Quebec French NLU Benchmark
+
+## Project overview
+COLE is a multidisciplinary Quebec French Natural Language Understanding benchmark with 23 tasks.
+- **Backend**: FastAPI (port 8000) — submission API, HuggingFace dataset evaluation
+- **Frontend**: Next.js 16 (port 8001) — leaderboard, submission form, bilingual (EN/FR)
+- **Deployment**: Docker container on HuggingFace Spaces (nginx on port 7860 proxies both)
+
+## Quick start
+```bash
+# Backend
+export HF_TOKEN=hf_...
+pip install -r cole/requirements.txt
+uvicorn cole.backend.submission_api:app --host 0.0.0.0 --port 8000
+
+# Frontend
+cd frontend && npm ci && npm run dev
+
+# Tests (requires HF_TOKEN for dataset access)
+pip install -r tests/tests_requirements.txt
+pytest
+```
+
+## Key directories
+- `cole/backend/` — FastAPI app (`submission_api.py` is the entry point)
+- `cole/dataset/` — HuggingFace dataset loading (repo: `graalul/COLE`)
+- `cole/task/` — Task definitions and evaluation logic
+- `cole/metrics/` — Metric wrappers (accuracy, F1, pearson, fquad); `math_metric.py` scores math answers by SymPy equivalence; `mixed_questions.py` evaluates JSONL files mixing several question types (Netquiz/COLE corpus)
+- `frontend/src/app/` — Next.js App Router pages and components
+- `tests/` — pytest tests (skip automatically without HF_TOKEN)
+
+## Commands
+```bash
+make lint        # Run pylint + eslint
+make format      # Check black formatting
+make test        # Run pytest
+make build       # Build frontend
+make docker      # Build Docker image
+make all         # lint + format + test + build
+```
+
+## CI/CD
+- **Formatting**: `black --check .` (Python 3.12)
+- **Linting**: `pylint cole/ tests/` (Python 3.10, 3.11, 3.12)
+- **Tests**: `pytest` (Python 3.12, requires HF_TOKEN secret)
+- **Frontend build**: `npm ci && npm run lint && npm run build`
+- **Docker build**: builds and validates the Docker image
+- **Deploy**: pushes to HuggingFace Spaces on main/dev push
+
+## Evaluating mixed-type question files
+The Netquiz/COLE corpus stores several question types in a single JSONL file, each
+row keeping its type in a `question_type` field. `cole.metrics.mixed_questions`
+picks the metric per row (reusing the `fquad_metric` primitives) and reports a
+per-type score plus two composites (unweighted, GLUE-style; and weighted by the
+instance count per type):
+```bash
+python -m cole.metrics.mixed_questions gold.jsonl
+python -m cole.metrics.mixed_questions gold.jsonl --predictions preds.jsonl --output report.json
+```
+Supported types and their metric: `single_choice`/`true_false` (accuracy),
+`multiple_choice` (set F1), `short_answer` (FQuAD EM+F1, or SymPy-based math
+equivalence via `math_metric.py` when the row's `subjects` flag it as
+mathematical), `association` (pair F1), `categorization` (per-element accuracy),
+`ordering` (exact match + pairwise concordance).
+
+## Conventions
+- Python: black formatting (line-length 88), pylint score must be 10.0
+- Frontend: ESLint with eslint-config-next flat config
+- All components use `'use client'` directive (client-side i18n)
+- Translations in `frontend/src/app/{en,fr}/translation.json`

CODE_OF_CONDUCT.md

@@ -0,0 +1,35 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+
+Examples of unacceptable behavior:
+
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information without explicit permission
+* Other conduct which could reasonably be considered inappropriate
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the project team at david.beauchemin@ift.ulaval.ca.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1.

Dockerfile

@@ -0,0 +1,55 @@
+# Stage 1: Build frontend
+FROM node:20-slim AS frontend-build
+WORKDIR /app/frontend
+COPY frontend/package*.json ./
+RUN npm ci
+COPY frontend/ ./
+RUN npm run build
+
+# Stage 2: Final image with backend + built frontend
+FROM python:3.12-slim
+
+WORKDIR /app
+
+# Install system dependencies (nginx, curl, Node.js runtime for Next.js)
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    nginx \
+    curl \
+    netcat-openbsd \
+    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
+    && apt-get install -y --no-install-recommends nodejs \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Python dependencies
+COPY cole/docker_requirements.txt /app/cole/
+RUN pip install --no-cache-dir --upgrade pip wheel \
+    && pip install --no-cache-dir --prefer-binary pyarrow pandas numpy scipy fsspec aiohttp tqdm \
+    && pip install --no-cache-dir --prefer-binary -r /app/cole/docker_requirements.txt
+
+# Copy backend source
+COPY cole/ /app/cole/
+
+# Copy built frontend from stage 1
+COPY --from=frontend-build /app/frontend /app/frontend
+
+# Copy config files
+COPY nginx.conf /etc/nginx/nginx.conf
+COPY start.sh /start.sh
+RUN chmod +x /start.sh
+
+# Create non-root user and set permissions
+RUN useradd -m -u 1000 user \
+    && mkdir -p /app/.cache /var/lib/nginx /var/log/nginx /app/logs /run \
+    && touch /run/nginx.pid \
+    && chown -R user:user /app /var/lib/nginx /var/log/nginx /run
+
+ENV HF_HOME=/app/.cache \
+    HF_DATASETS_CACHE=/app/.cache
+
+USER user
+EXPOSE 7860
+
+HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
+  CMD curl -f http://localhost:7860/ || exit 1
+
+CMD ["sh", "/start.sh"]

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 GRAAL Research Group, Université Laval
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -0,0 +1,19 @@
+.PHONY: lint format test build docker all
+
+lint:
+	pylint cole/ tests/
+	cd frontend && npm run lint
+
+format:
+	black --check .
+
+test:
+	pytest
+
+build:
+	cd frontend && npm ci && npm run build
+
+docker:
+	docker build -t cole .
+
+all: format lint test build

README.md

@@ -0,0 +1,128 @@
+---
+title: COLE !
+emoji: 🐳
+colorFrom: purple
+colorTo: gray
+sdk: docker
+app_port: 7860
+---
+
+# COLE: Comprehensive Benchmark for Quebec French Language Understanding Evaluation
+
+[![Website](https://img.shields.io/badge/Website-colebenchmark.org-blue)](https://colebenchmark.org/)
+[![Paper](https://img.shields.io/badge/Paper-arXiv%3A2510.05046-b31b1b)](https://arxiv.org/abs/2510.05046)
+[![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ffd21e)](https://huggingface.co/datasets/graalul/COLE-public)
+[![Coverage](https://raw.githubusercontent.com/GRAAL-Research/COLE/badges/coverage-badge.svg)](https://github.com/GRAAL-Research/COLE/actions)
+
+**COLE** is a comprehensive benchmark for evaluating Quebec French Natural Language Understanding (NLU). It includes 23 diverse tasks covering sentiment analysis, paraphrase detection, natural language inference, question answering, grammatical judgment, word sense disambiguation, and more — with a particular focus on linguistic phenomena relevant to the French language.
+
+We benchmark 94 large language models (LLMs), providing an extensive analysis of the current state of Quebec French NLU. Our results highlight a significant performance gap between closed- and open-weight models and identify key challenging frontiers such as zero-shot extractive question answering, fine-grained word sense disambiguation, and understanding of regional language variations.
+
+## Links
+
+- **Leaderboard**: [colebenchmark.org](https://colebenchmark.org/)
+- **Paper**: [COLE: a Comprehensive Benchmark for Quebec French Language Understanding Evaluation (arXiv:2510.05046)](https://arxiv.org/abs/2510.05046)
+- **Dataset**: [HuggingFace — graalul/COLE-public](https://huggingface.co/datasets/graalul/COLE-public)
+
+## Tasks
+
+COLE consists of 23 tasks grouped by NLU capability:
+
+### Sentiment Analysis
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **Allocine** | Sentiment classification of French movie reviews (positive/negative) | 20,000 |
+| **MMS-fr** | Sentiment analysis with 3 classes (positive, neutral, negative) | 63,190 |
+
+### Natural Language Inference (NLI)
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **DACCORD** | Semantic plausibility / contradiction detection of French sentences (binary) | 1,034 |
+| **FraCaS** | NLI involving quantifiers, plurality, anaphora, and ellipsis | 346 |
+| **GQNLI-fr** | NLI with quantifier logic (e.g., most, at least, more than half) | 30 |
+| **LingNLI** | NLI corpus constructed with a linguist in the loop | 4,893 |
+| **MNLI-nineeleven-Fr-MT** | French machine-translated MNLI using 9/11 context | 2,000 |
+| **RTE3-Fr** | French version of RTE3 for textual entailment | 3,121 |
+| **SICK-fr** | Sentence pair relatedness and entailment | 4,906 |
+| **XNLI-fr** | Cross-lingual NLI in French | 5,010 |
+
+### Question Answering
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **FQuAD** | Extractive QA on high-quality French Wikipedia articles | 400 |
+| **Fr-BoolQ** | Boolean question answering in French | 178 |
+| **PIAF** | French extractive QA pairs | 384 |
+
+### Paraphrase Detection
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **PAWS-X** | Paraphrase identification from sentence pairs | 2,000 |
+| **QFrBLiMP** | Semantic equivalence detection between sentence pairs | 2,290 |
+
+### Grammatical Judgment
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **MultiBLiMP-Fr** | Grammatical correctness from minimal pairs | 77 |
+| **QFrCoLA** | Sentence acceptability in French (grammar, syntax) | 7,546 |
+
+### Semantic Similarity
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **STS22** | Document-level similarity of multilingual news articles | 72 |
+
+### Word Sense Disambiguation
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **WSD-Fr** | Disambiguating verb meanings in context | 3,121 |
+
+### Quebec French
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **QFrCoRE** | Matching Quebec French expressions to standard definitions | 4,633 |
+| **QFrCoRT** | Matching Quebec French terms to standard definitions | 201 |
+
+### Coreference / Pronoun Resolution
+| Task | Description | Test size |
+|------|-------------|-----------|
+| **Wino-X-LM** | Pronoun resolution with ambiguous referents | 2,793 |
+| **Wino-X-MT** | Translation-based pronoun resolution with gendered pronouns | 2,988 |
+
+## Language
+
+All data in COLE is in **French**.
+
+## Evaluating mixed-type question files
+
+Some corpora (e.g. the Netquiz/COLE corpus) store several question types in a single
+JSONL file, each row keeping its type in a `question_type` field. Since the right
+metric differs from one question to the next, `cole.metrics.mixed_questions` selects
+the metric per row and aggregates the results:
+
+```bash
+# Predictions embedded in each row's "prediction" field
+python -m cole.metrics.mixed_questions gold.jsonl
+
+# Predictions in a separate file aligned line by line, JSON report written out
+python -m cole.metrics.mixed_questions gold.jsonl --predictions preds.jsonl --output report.json
+```
+
+It reuses COLE's metric primitives and reports a per-type score plus two composite
+scores: an unweighted mean of the per-type scores (GLUE-style) and a mean weighted by
+the number of instances of each type. Supported types: `single_choice`, `true_false`,
+`multiple_choice`, `short_answer`, `association`, `categorization`, `ordering`.
+Mathematical `short_answer` rows (flagged via their `subjects`) are scored with
+SymPy-based equivalence so answers like `2^5` and `32` or `1/2` and `0.5` match.
+
+## Citation
+
+If you use COLE in your research, please cite our paper:
+
+```bibtex
+@article{beauchemin2025cole,
+  title={COLE: a Comprehensive Benchmark for Quebec French Language Understanding Evaluation},
+  author={Beauchemin, David and Tremblay, Yan and Youssef, Mohamed Amine and Khoury, Richard},
+  journal={arXiv preprint arXiv:2510.05046},
+  year={2025},
+  url={https://arxiv.org/abs/2510.05046}
+}
+```

SECURITY.md

@@ -0,0 +1,15 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 1.0.x   | :white_check_mark: |
+
+## Reporting a Vulnerability
+
+If you discover a security vulnerability, please report it responsibly by emailing **david.beauchemin@ift.ulaval.ca**.
+
+Please do **not** open a public GitHub issue for security vulnerabilities.
+
+We will acknowledge your report within 48 hours and provide a timeline for a fix.

cole/__init__.py

@@ -0,0 +1,6 @@
+REPO_ID = "COLE-Graal/COLEGraal"
+cole = "COLE-final"
+boreal = "COLE-final-boreal"
+complete = "COLE-finale-complete"
+comparison = "Fr-comparison"
+NA_VALUE = -1

cole/backend/evaluation.py

@@ -0,0 +1,41 @@
+import copy
+import operator
+from functools import reduce
+from typing import List, Dict
+
+from cole.task.task_factory import Task
+
+
+def compute_tasks_ratings(tasks: List[Task], submission: Dict) -> Dict:
+    """
+    Method to compute the tasks ratings.
+    :param tasks: list of tasks
+    :param submission: submission dictionary
+    """
+
+    # We merge the tasks dictionary for simpler handling.
+    submission_copy = copy.deepcopy(submission)
+    submission_response = reduce(operator.ior, submission_copy.get("tasks"), {})
+
+    for task in tasks:
+        task_name = task.task_name
+
+        # We remove the prediction since we do not keep it in the response.
+        # Validation allows both "predictions" and "prediction"; pop whichever is present.
+        task_payload = submission_response.get(task_name)
+        if "predictions" in task_payload:
+            predictions = task_payload.pop("predictions")
+        else:
+            predictions = task_payload.pop("prediction")
+
+        ratings, warning = task.compute(predictions=predictions)
+        ratings.update({f"{task.metric_name}_warning": warning})
+        submission_response.get(task_name).update({f"{task.metric_name}": ratings})
+
+    # Final submission response where we unwrap the merge tasks dictionary into a list of dictionary.
+    submission_response = {
+        "model_name": submission.get("model_name"),
+        "model_url": submission.get("model_url"),
+        "tasks": [{key: value} for key, value in submission_response.items()],
+    }
+    return submission_response

cole/backend/submission_api.py

@@ -0,0 +1,251 @@
+import glob
+import json
+import logging
+import os
+import uuid
+from contextlib import asynccontextmanager
+from datetime import datetime
+from functools import lru_cache
+from pathlib import Path
+from typing import Dict, List, Any, Union
+
+import huggingface_hub
+from fastapi import FastAPI, UploadFile, Form, File, HTTPException
+from fastapi.responses import JSONResponse
+from fastapi.staticfiles import StaticFiles
+
+from slowapi import Limiter
+from slowapi.util import get_remote_address
+from slowapi.errors import RateLimitExceeded
+from starlette.middleware.cors import CORSMiddleware
+from starlette.requests import Request
+
+from cole.backend.evaluation import compute_tasks_ratings
+from cole.backend.submit_tools import unzip_predictions_from_zip
+from cole.dataset.datasets_data import preload_all_datasets
+from cole.backend.validation_tools import (
+    validate_submission_tasks_name,
+    validate_submission_json,
+    validate_submission_template,
+)
+from cole.task.task import Task
+from cole.task.task_factory import (
+    tasks_factory,
+)
+
+MAX_ZIP_SIZE_MB = 50
+
+BASE_DIR = Path(__file__).resolve().parents[2]
+RESULTS_DIR = BASE_DIR / "cole" / "backend" / "results"
+RESULTS_DIR.mkdir(parents=True, exist_ok=True)
+FRONTEND_DIR = BASE_DIR / "frontend"
+
+
+@asynccontextmanager
+async def lifespan(application: FastAPI = None):  # pylint: disable=unused-argument
+    """Called before the backend comes online, is used to load datasets in memory."""
+    # Load the ML model
+    try:
+        token = os.environ.get("HF_TOKEN")
+        huggingface_hub.login(token=token)
+        preload_all_datasets()
+    except Exception as e:
+        error_message = f"The datasets could not be loaded : {e}"
+        logging.critical(error_message)
+
+    yield
+
+
+limiter = Limiter(key_func=get_remote_address)
+app = FastAPI(lifespan=lifespan)
+app.state.limiter = limiter
+app.add_exception_handler(
+    RateLimitExceeded,
+    lambda req, exc: JSONResponse(
+        status_code=429,
+        content={"detail": "Too many submissions. Please try again later."},
+    ),
+)
+app.mount("/results", StaticFiles(directory=str(RESULTS_DIR)), name="results")
+front_end_info_message = f"The Front-end directory is: {FRONTEND_DIR}"
+logging.info(front_end_info_message)
+
+ALLOWED_ORIGINS = os.environ.get(
+    "CORS_ORIGINS",
+    "https://davebulaval-cole.hf.space,http://localhost:3000,http://localhost:8001",
+).split(",")
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=ALLOWED_ORIGINS,
+    allow_methods=["GET", "POST"],
+    allow_headers=["*"],
+)
+
+
+@app.post("/submit")
+@limiter.limit("5/minute")
+async def submit(
+    request: Request,  # pylint: disable=unused-argument  # required by slowapi limiter
+    email: str = Form(...),
+    predictions_zip: UploadFile = File(...),
+    display_name: str = Form(...),
+):
+    """Route for making submissions with user generated results.
+    :param request : The incoming request (used for rate limiting)
+    :param email : The email of the user's submission
+    :param predictions_zip : The zip file of the user's predictions'
+    :param display_name : The display name associated with the user's submission'
+    """
+    logging.info("Starting submission")
+    if len(display_name) > 200:
+        raise HTTPException(
+            status_code=400, detail="Display name must be under 200 characters."
+        )
+    if len(email) > 320 or "@" not in email:
+        raise HTTPException(status_code=400, detail="Invalid email address.")
+    info_message = f"Submission from {email!r} as {display_name!r}."
+    logging.info(info_message)
+    zip_bytes = await predictions_zip.read()
+    if len(zip_bytes) > MAX_ZIP_SIZE_MB * 1024 * 1024:
+        raise HTTPException(
+            status_code=413, detail=f"ZIP file exceeds {MAX_ZIP_SIZE_MB}MB limit."
+        )
+    submission_json = unzip_predictions_from_zip(zip_bytes)
+
+    validate_submission_template(submission_json)
+    validate_submission_tasks_name(submission_json)
+    validate_submission_json(submission_json)
+
+    tasks: List[Task] = tasks_factory(submission_json)
+    logging.info("Computation started")
+    start = datetime.now()
+    submission_response = compute_tasks_ratings(tasks=tasks, submission=submission_json)
+    computation_time = datetime.now() - start
+    info_message = f"Computation ended in {computation_time}"
+    logging.info(info_message)
+    submission_id = str(uuid.uuid4())
+    submission_response.update(
+        {
+            "display_name": display_name,
+            "email": email,
+            "submission_id": submission_id,
+        }
+    )
+
+    out_path = RESULTS_DIR / f"{submission_id}.json"
+    with open(out_path, "w", encoding="utf-8") as f:
+        json.dump(submission_response, f, ensure_ascii=False, indent=2)
+
+    get_leaderboard_entries.cache_clear()
+
+    return JSONResponse(content=submission_response)
+
+
+@lru_cache(maxsize=1)
+def get_leaderboard_entries() -> List[Dict[str, Any]]:
+    """Returns all entries currently in the leaderboard.
+    Supporte aussi les fichiers JSON qui contiennent une LISTE d'entrées
+    et normalise les métriques 'plates' en groupes imbriqués pour le front.
+    """
+
+    def _wrap_flat_metrics(task_payload: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Si task_payload est 'plat' (ex: {"accuracy": 94.2}),
+        on le transforme en {"<group>": {...}} pour que le front puisse l'agréger.
+        Règles de nommage du groupe :
+          - présence de exact_match/f1 -> "fquad"
+          - sinon présence de acc/accuracy -> "accuracy"
+          - sinon présence de pearson/pearsonr/spearman -> "correlation"
+          - sinon -> "metrics"
+        Les valeurs >1 sont laissées telles quelles (le front normalise déjà % -> [0,1]).
+        """
+        if not isinstance(task_payload, dict):
+            return task_payload
+
+        # si c'est déjà "imbriqué" (une valeur est un dict), on ne touche pas
+        if any(isinstance(v, dict) for v in task_payload.values()):
+            return task_payload
+
+        keys = set(k.lower() for k in task_payload.keys())
+        if {"exact_match", "f1"} & keys:
+            group = "fquad"
+        elif {"accuracy", "acc"} & keys:
+            group = "accuracy"
+        elif {"pearson", "pearsonr", "spearman"} & keys:
+            group = "correlation"
+        else:
+            group = "metrics"
+
+        # Rien de spécial pour les warnings ici : le front les considère optionnels
+        # et s'attend à "<group>_warning" dans l'objet interne si on veut en fournir.
+        return {group: task_payload}
+
+    entries: List[Dict[str, Any]] = []
+
+    for filepath in glob.glob(str(RESULTS_DIR / "*.json")):
+        try:
+            with open(filepath, encoding="utf-8") as f:
+                data = json.load(f)
+
+            # Fonction interne qui traite UNE entrée (dict) au bon format minimal
+            def process_entry(entry: Dict[str, Any]) -> Union[Dict[str, Any], None]:
+                if not isinstance(entry, dict):
+                    return None
+                if "model_name" not in entry or "tasks" not in entry:
+                    return None
+
+                # Re-construire "results" comme le front s'y attend
+                results = {}
+                for task_obj in entry.get("tasks", []):
+                    if not isinstance(task_obj, dict) or len(task_obj) != 1:
+                        continue
+                    task_name, payload = list(task_obj.items())[0]
+                    normalized = _wrap_flat_metrics(payload)
+                    results[task_name] = normalized
+
+                if not results:
+                    return None
+
+                return {
+                    "submission_id": entry.get("submission_id") or str(uuid.uuid4()),
+                    "display_name": entry.get("display_name")
+                    or entry.get("model_name")
+                    or "Unnamed Model",
+                    "email": entry.get("email", "N/A"),
+                    "results": results,
+                }
+
+            # Le fichier peut contenir UNE entrée (dict) ou PLUSIEURS (list)
+            if isinstance(data, list):
+                for item in data:
+                    processed = process_entry(item)
+                    if processed:
+                        entries.append(processed)
+            else:
+                processed = process_entry(data)
+                if processed:
+                    entries.append(processed)
+
+        except Exception as e:
+            logging_message = f"Error processing file '{filepath}': {e}"
+            logging.error(logging_message)
+            continue
+
+    return entries
+
+
+@app.get("/leaderboard")
+async def leaderboard() -> List[Dict[str, Any]]:
+
+    return get_leaderboard_entries()
+
+
+@app.get("/health")
+async def health_check():
+    return {"status": "healthy", "message": "API is running."}
+
+
+@app.get("/")
+async def home():
+    return {"status": "working"}

cole/backend/submit_tools.py

@@ -0,0 +1,37 @@
+import io
+import json
+import zipfile
+
+from fastapi import HTTPException
+
+MAX_DECOMPRESSED_SIZE_MB = 200
+
+
+def unzip_predictions_from_zip(zip_bytes: bytes) -> dict:
+    """
+    Reads predictions.json directly from the ZIP in memory.
+    """
+    try:
+        zip_file = zipfile.ZipFile(io.BytesIO(zip_bytes))
+    except zipfile.BadZipFile as exc:
+        raise HTTPException(
+            400, "The uploaded file is not a valid ZIP archive."
+        ) from exc
+
+    with zip_file as z:
+        if "predictions.json" not in z.namelist():
+            error_message = (
+                "The uploaded ZIP file does not contains a predictions.json file."
+            )
+            raise HTTPException(400, error_message)
+        info = z.getinfo("predictions.json")
+        if info.file_size > MAX_DECOMPRESSED_SIZE_MB * 1024 * 1024:
+            raise HTTPException(
+                413,
+                f"Decompressed predictions.json exceeds {MAX_DECOMPRESSED_SIZE_MB}MB limit.",
+            )
+        with z.open("predictions.json") as f:
+            try:
+                return json.load(f)
+            except json.JSONDecodeError as exc:
+                raise HTTPException(400, "predictions.json is not valid JSON.") from exc

cole/backend/validation_tools.py

@@ -0,0 +1,90 @@
+import logging
+from typing import Dict, List
+
+from fastapi import HTTPException
+
+from cole.task.task_names import Tasks
+
+tasks_name = [task.value for task in Tasks]
+
+
+def validate_submission_template(dictionary: Dict) -> None:
+    """Ensures the dictionnary follows the correct format.
+    :param dictionary: Dictionary to validate."""
+    if dictionary.get("model_name", None) is None:
+        error = "The submission is missing a model name."
+        logging.error(error)
+        raise HTTPException(400, error)
+    if dictionary.get("model_url", None) is None:
+        error = "The submission is missing a model URL."
+        logging.error(error)
+        raise HTTPException(400, error)
+    if dictionary.get("tasks", None) is None:
+        error = "The submission is missing a tasks keyword."
+        logging.error(error)
+        raise HTTPException(400, error)
+
+    tasks = dictionary.get("tasks")
+    if not isinstance(tasks, List):
+        error = (
+            "The tasks keyword value must be a list of dictionaries where they key is the tasks "
+            "and value is a dictionary of predictions (in a list format). See our documentation for"
+            "a template."
+        )
+        logging.error(error)
+        raise HTTPException(400, error)
+
+    for task in tasks:
+        if not isinstance(task, dict) or len(task.keys()) != 1:
+            error = (
+                "Each task must be a dictionary of one element where the key is "
+                "the task name and the value is a list."
+            )
+            logging.error(error)
+            raise HTTPException(400, error)
+
+
+def validate_submission_tasks_name(dictionary: Dict) -> None:
+    """
+    Validate if the submission JSON key are the tasks name.
+    """
+    for task in dictionary.get("tasks"):
+        key = list(task.keys())[0]
+        if key not in tasks_name:
+            error = f"Unknown key '{key}' in the submission JSON. The expected tasks are: {tasks_name}."
+            logging.error(error)
+            raise HTTPException(400, error)
+
+
+def validate_submission_json(dictionary: Dict) -> None:
+    """Validates that the submitted json is in the correct format.
+    :param dictionary: Dictionary to validate."""
+    task_payload = dictionary.get("tasks")
+
+    for task in task_payload:
+        for task_name, payload in task.items():
+            if not isinstance(payload, dict):
+                error = (
+                    "The tasks payload must be a dictionary in the format '{'prediction': [<predictions>]}' "
+                    "for each task."
+                )
+                logging.error(error)
+                raise HTTPException(400, error)
+            if not ({"predictions", "prediction"} & payload.keys()):
+                # Empty payload (`{}`) used to slip through and crash compute_tasks_ratings
+                # with a KeyError -> HTTP 500. Reject it explicitly here.
+                error = f"The task '{task_name}' payload does not have the expected key: 'predictions'."
+                logging.error(error)
+                raise HTTPException(400, error)
+            for key, value in payload.items():
+                if key not in ["predictions", "prediction"]:
+                    error = f"The task '{task_name}' payload does not have the expected key: 'predictions'."
+                    logging.error(error)
+                    raise HTTPException(400, error)
+                if not isinstance(value, list):
+                    error = (
+                        f"The task '{task_name}' predictions payload is not in a list format. "
+                        r"The expected format is: '{'prediction': [<predictions>]}'"
+                    )
+                    logging.error(error)
+                    raise HTTPException(400, error)

cole/dataset/dataset.py

@@ -0,0 +1,101 @@
+from typing import Callable, Any, Union, List
+
+from datasets import load_dataset
+
+
+class Dataset:
+    """Class representing a usable dataset.
+    Allows dataset to be expressed as multiple forms, including as prompts, data or answers.
+    :param name : name of the dataset.
+    :param description : description of the dataset.
+    :param possible_ground_truths : the form that could be taken by ground truths.
+    :param hugging_face_repo : where to download the dataset on HuggingFace.
+    :param line_to_truth_fn : a function converting a dataset line to its truth value.
+    :param line_to_prompt_fn : a function converting a dataset line to a prompt for LLM inference.
+    :param line_to_data_fn : a function converting a dataset line to its data value for non LLM inference.
+    """
+
+    def __init__(
+        self,
+        name: str,
+        description: str,
+        possible_ground_truths: Union[List[str], List[int], List[float]],
+        hugging_face_repo: str,
+        line_to_truth_fn: Callable,
+        line_to_prompt_fn: Callable,
+        line_to_data_fn: Callable,
+    ):
+        self._dataset = None
+        self._ground_truths_cache = None
+        self.name = name
+        self.description = description
+        self.hugging_face_repo = hugging_face_repo
+        self.possible_ground_truths = possible_ground_truths
+        self.line_to_prompt_fn = line_to_prompt_fn
+        self.line_to_truth_fn = line_to_truth_fn
+        self.line_to_data_fn = line_to_data_fn
+
+    @property
+    def dataset(self):
+        self.load_data()
+        return self._dataset
+
+    def load_data(self):
+        if self._dataset is None:
+            self._dataset = load_dataset(
+                self.hugging_face_repo, name=self.name, split="test"
+            )
+
+    @property
+    def ground_truths(self) -> Union[List[str], List[int], List[float]]:
+        """The dataset's ground truths as a list (cached after first computation)"""
+        if self._ground_truths_cache is None:
+            self._ground_truths_cache = [
+                self.line_to_truth_fn(line) for line in self.dataset
+            ]
+        return self._ground_truths_cache
+
+    @property
+    def prompts(self) -> List[str]:
+        """The dataset's prompts as a list"""
+        return [self.line_to_prompt_fn(line) for line in self.dataset]
+
+    @property
+    def data(self) -> List[str]:
+        """The dataset's data as a list"""
+        return [self.line_to_data_fn(line) for line in self.dataset]
+
+    @property
+    def metadata(self) -> dict[str, Any]:
+        """The dataset's metadata as a dict"""
+        return {
+            "name": self.name,
+            "description": self.description,
+            "possible_ground_truths": str(self.possible_ground_truths),
+            "Prompt template": self.line_to_prompt_fn(self.EchoDict()),
+        }
+
+    @property
+    def metadata_string(self) -> str:
+        """The dataset's metadata as a string"""
+        lines = []
+        for key, value in self.metadata.items():
+            lines.append(f"{key}: {value}")
+        return "\n".join(lines)
+
+    def __len__(self):
+        return len(self.dataset)
+
+    def __getitem__(self, index: Union[int, slice]):
+        if isinstance(index, slice):
+            get_item_data = self.ground_truths[index.start : index.stop]
+        else:
+            get_item_data = self.ground_truths[index]
+
+        return get_item_data
+
+    class EchoDict:
+        """Helper class for building prompt templates,always returns the accessed key"""
+
+        def __getitem__(self, key):
+            return key

cole/dataset/datasets_data.py

@@ -0,0 +1,797 @@
+from cole.dataset.dataset import Dataset
+from cole.dataset.prompt_builder import PromptBuilder
+from cole.task import COLE_REPOSITORY_NAME
+from cole.task.task_names import COLETasks, BorealTasks
+
+datasets = {
+    COLETasks.ALLOCINE.value: Dataset(
+        name=COLETasks.ALLOCINE.value,
+        description="Binary classification on sentiment analysis"
+        " of movie reviews, with reviews being either positive (1) or negative (0).",
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise("Cette phrase possède-t-elle un sentiment positif ou négatif ?")
+        .add_data(line["review"])
+        .add_end(
+            (
+                "Réponds "
+                "uniquement par 1 si la phrase est positive, réponds par 0 sinon. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: line["review"],
+    ),
+    COLETasks.QFRCOLA.value: Dataset(
+        name=COLETasks.QFRCOLA.value,
+        description="Binary grammatical judgement : "
+        "Predicts whether a sentence is grammatically correct (1) or not. (0).",
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise("Juge si cette phrase est grammaticalement correcte :")
+        .add_data(line["sentence"])
+        .add_end(
+            (
+                "Réponds avec seulement 1 si la phrase est grammaticalement correcte, 0 sinon. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: line["sentence"],
+    ),
+    COLETasks.QFRBLIMP.value: Dataset(
+        name=COLETasks.QFRBLIMP.value,
+        description="Choice task between two sentences : Choose the one which is grammatically correct.",
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(
+            line["label"]
+        ),  # The label is return as a string.
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise("Laquelle de ces phrases est grammaticalement correcte ?")
+            .add_data(f"Phrase 0:{line['sentence_a']}")
+            .add_data(f"Phrase 1:{line['sentence_b']}")
+            .add_end(
+                "Réponds avec seulement 0 si la phrase 0 "
+                "est grammaticalement correcte, et uniquement 1 si la phrase 1 est grammaticalement "
+                "correcte. La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {line["sentence_a"], line["sentence_b"]},
+    ),
+    COLETasks.GQNLI.value: Dataset(
+        name=COLETasks.GQNLI.value,
+        description="Natural language inference task : "
+        "predict the relation between two sentences (implication, neutral, contradiction).",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Quelle est la relation de la deuxième phrase par rapport à la première ?"
+        )
+        .add_data(line["premise"])
+        .add_data(line["hypothesis"])
+        .add_end(
+            (
+                "Réponds uniquement par :\n"
+                "0 - si la deuxième phrase implique la première,\n"
+                "1 - si la relation est neutre,\n"
+                "2 - s'il y a contradiction.\n"
+                "Réponds uniquement par 0, 1 ou 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "premise": line["premise"],
+            "hypothesis": line["hypothesis"],
+        },
+    ),
+    COLETasks.SICKFR.value: Dataset(
+        name=COLETasks.SICKFR.value,
+        description="Natural language inference task : "
+        "predict the relation between two sentences (implication, neutral, contradiction).",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["label"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise("Détermine la relation entre les deux phrases suivantes :")
+        .add_data(f"Phrase A : {line['sentence_A']}\nPhrase B : {line['sentence_B']}")
+        .add_end(
+            "Réponds uniquement par 0, 1 ou 2 :\n"
+            "0 - si la deuxième phrase découle logiquement de la première,\n"
+            "1 - si leur relation est neutre,\n"
+            "2 - si les phrases se contredisent.\n"
+            "La réponse est :"
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "sentence_A": line["sentence_A"],
+            "sentence_B": line["sentence_B"],
+        },
+    ),
+    COLETasks.STS22.value: Dataset(
+        name=COLETasks.STS22.value,
+        description="Semantic textual similarity task : "
+        "Predict how similar two sentences are to each other (1 to 4).",
+        possible_ground_truths=["1", "2", "3", "4"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["score"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "À quel point les deux phrases suivantes sont-elles similaires ? Donne une note entière de 1 à 4."
+        )
+        .add_data(f"Phrase 1 : {line['sentence1']}\nPhrase 2 : {line['sentence2']}")
+        .add_end(
+            "Réponds uniquement avec un nombre entier entre 1 (aucune similarité) et 4 (équivalence parfaite). "
+            "La réponse est :"
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "sentence1": line["sentence1"],
+            "sentence2": line["sentence2"],
+        },
+    ),
+    COLETasks.PAWS_X.value: Dataset(
+        name=COLETasks.PAWS_X.value,
+        description="Binary classification task : "
+        "Predict if two sentences have the same meaning (1) or not (0).",
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Les deux phrases suivantes veulent-elles dire la même chose, ou ont-elles des significations différentes ?"
+        )
+        .add_data(line["sentence1"])
+        .add_data(line["sentence2"])
+        .add_end(
+            (
+                "Réponds seulement 1 si les deux phrases ont la même signification, 0 sinon. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "sentence1": line["sentence1"],
+            "sentence2": line["sentence2"],
+        },
+    ),
+    COLETasks.PIAF.value: Dataset(
+        name=COLETasks.PIAF.value,
+        description="Extractive question answering task : Extract a question's answer from a given context.",
+        possible_ground_truths=[],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["answers"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Tu vas recevoir un contexte suivi d'une question.\n"
+            "Ta tâche est d'extraire **mot pour mot** le passage du contexte qui répond le mieux à la question.\n"
+            "N'invente rien. Ne reformule pas.\n"
+            "Réponds **en copiant uniquement** un extrait exact du texte ci-dessus."
+        )
+        .add_data(f"Contexte  : {line['context']}")
+        .add_data(f"Question : {line['question']}")
+        .add_end(
+            "Réponds uniquement par un passage extrait du contexte. La réponse est :"
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "context": line["context"],
+            "question": line["question"],
+        },
+    ),
+    COLETasks.FQUAD.value: Dataset(
+        name=COLETasks.FQUAD.value,
+        description="Extractive question answering task : Extract a question's answer from a given context.",
+        possible_ground_truths=[],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["answers"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Tu vas recevoir un contexte suivi d'une question.\n"
+            "Ta tâche est d'extraire **mot pour mot** le passage du contexte qui répond le mieux à la question.\n"
+            "N'invente rien. Ne reformule pas.\n"
+            "Réponds **en copiant uniquement** un extrait exact du texte ci-dessus."
+        )
+        .add_data(f"Contexte  : {line['context']}")
+        .add_data(f"Question : {line['question']}")
+        .add_end(
+            "Réponds uniquement par un passage extrait du contexte. La réponse est :"
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "context": line["context"],
+            "question": line["question"],
+        },
+    ),
+    COLETasks.XNLI.value: Dataset(
+        name=COLETasks.XNLI.value,
+        description="Natural language inference task : "
+        "predict the relation between two sentences (implication, neutral, contradiction).",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["label"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Quelle est la relation de la deuxième phrase par rapport à la première ?"
+        )
+        .add_data(rf"premise : {line['premise']}\n" f"sentence 2: {line['hypothesis']}")
+        .add_end(
+            (
+                "Réponds uniquement par :\n"
+                "0 - si la deuxième phrase implique la première,\n"
+                "1 - si la relation est neutre,\n"
+                "2 - s'il y a contradiction.\n"
+                "Réponds uniquement par 0, 1 ou 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "premise": line["premise"],
+            "hypothesis": line["hypothesis"],
+        },
+    ),
+    COLETasks.QFRCORE.value: Dataset(
+        name=COLETasks.QFRCORE.value,
+        description="Definition matching task : "
+        "Match the Quebec expression with its definition from a list.",
+        possible_ground_truths=[str(i) for i in range(10)],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["correct_index"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            f"Que veut dire cette expression québécoise « {line['expression']} » ?"
+        )
+        .add_data(
+            "\n".join(
+                f"{idx} - {definition}"
+                for idx, definition in enumerate(line["choices"])
+            )
+        )
+        .add_end(
+            (
+                "Réponds uniquement par l'index, débutant à zéro,  "
+                "de la bonne définition parmi la liste ci-dessus. Par exemple, si la "
+                "troisième phrase correspond à l'expression, la réponse sera 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "expression": line["expression"],
+            "choices": line["choices"],
+        },
+    ),
+    COLETasks.QFRCORT.value: Dataset(
+        name=COLETasks.QFRCORT.value,
+        description="Definition matching task : "
+        "Match the Quebec term with its definition from a list.",
+        possible_ground_truths=[str(i) for i in range(10)],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["correct_index"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            f"Qu'est-ce que ça veut dire ce terme québécois « {line['terme']} » ?"
+        )
+        .add_data(
+            "\n".join(
+                f"{idx} - {definition}"
+                for idx, definition in enumerate(line["choices"])
+            )
+        )
+        .add_end(
+            (
+                "Réponds uniquement par l'index, débutant à zéro,  "
+                "de la bonne définition parmi la liste ci-dessus. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "terme": line["terme"],
+            "choices": line["choices"],
+        },
+    ),
+    COLETasks.FRCOE.value: Dataset(
+        name=COLETasks.FRCOE.value,
+        description="Definition matching task : "
+        "Match the French expression with its definition from a list.",
+        possible_ground_truths=[str(i) for i in range(10)],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["correct_index"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            f"Que veut dire cette expression française « {line['expression']} » ?"
+        )
+        .add_data(
+            "\n".join(
+                f"{idx} - {definition}"
+                for idx, definition in enumerate(line["choices"])
+            )
+        )
+        .add_end(
+            (
+                "Réponds uniquement par l'index, débutant à zéro,  "
+                "de la bonne définition parmi la liste ci-dessus. Par exemple, si la "
+                "troisième phrase correspond à l'expression, la réponse sera 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "expression": line["expression"],
+            "choices": line["choices"],
+        },
+    ),
+    COLETasks.DACCORD.value: Dataset(
+        name=COLETasks.DACCORD.value,
+        description="Paraphrase detection task :"
+        "Predict whether the two sentences are compatible (0) "
+        "or contradict each other (1).",
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["label"]),
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise("Détermine la relation entre les deux phrases suivantes :")
+            .add_data(f"Première phrase : {line['premise']}")
+            .add_data(f"Deuxième phrase : {line['hypothesis']}")
+            .add_end(
+                "Réponds uniquement par :\n"
+                "0 - si les deux phrases sont compatibles (elles expriment la même information ou sont cohérentes),\n"
+                "1 - s'il y a contradiction entre les deux phrases.\n"
+                "Réponds uniquement par 0 ou 1. La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "premise": line["premise"],
+            "hypothesis": line["hypothesis"],
+        },
+    ),
+    COLETasks.FRENCH_BOOLQ.value: Dataset(
+        name=COLETasks.FRENCH_BOOLQ.value,
+        description="Binary question answering task : "
+        "Answer whether the context allows answering 'yes' to the question (1)"
+        "or, if the context only allows answering 'no' "
+        "to the question or does not answer the question. (0).",
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["label"]),
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise(
+                "Lis le passage suivant et réponds à la question en te basant uniquement sur le texte :\n"
+                "- Si le passage permet d'affirmer que la réponse à la question est oui, réponds 1.\n"
+                "- Sinon, si la réponse est non ou que le passage ne permet pas de répondre à la question, réponds 0."
+            )
+            .add_data(f"Passage : {line['passage']}")
+            .add_data(f"Question : {line['question']}")
+            .add_end("La réponse est :")
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "question": line["question"],
+            "passage": line["passage"],
+        },
+    ),
+    COLETasks.MNLI_NINEELEVEN_FR_MT.value: Dataset(
+        name=COLETasks.MNLI_NINEELEVEN_FR_MT.value,
+        description="Natural language inference task : "
+        "predict the relation between two sentences (implication, neutral, contradiction).",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Quelle est la relation de la deuxième phrase par rapport à la première ?"
+        )
+        .add_data(line["premise"])
+        .add_data(line["hypothesis"])
+        .add_end(
+            (
+                "Réponds uniquement par :\n"
+                "0 - si la deuxième phrase implique la première,\n"
+                "1 - si la relation est neutre,\n"
+                "2 - s'il y a contradiction.\n"
+                "Réponds uniquement par 0, 1 ou 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "premise": line["premise"],
+            "hypothesis": line["hypothesis"],
+        },
+    ),
+    COLETasks.RTE3_FRENCH.value: Dataset(
+        name=COLETasks.RTE3_FRENCH.value,
+        description="Natural language inference task : "
+        "predict the relation between two sentences (entailment, neutral, contradiction)",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["label"]),
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise(
+                "Lis le texte suivant et détermine la relation de l'énoncé par rapport au texte."
+            )
+            .add_data(f"Texte : {line['premise']}")
+            .add_data(f"Énoncé : {line['hypothesis']}")
+            .add_end(
+                "Réponds uniquement par 0, 1 ou 2 :\n"
+                "0 - si l'énoncé découle logiquement du texte (entailment),\n"
+                "1 - si la relation est neutre,\n"
+                "2 - s'il y a contradiction.\n"
+                "La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "premise": line["premise"],
+            "hypothesis": line["hypothesis"],
+        },
+    ),
+    COLETasks.WINO_X_LM.value: Dataset(
+        name=COLETasks.WINO_X_LM.value,
+        description=(
+            "Pronoun resolution task : predict the correct referent (1 or 2) "
+            "of a pronoun in a sentence by choosing between two candidates."
+        ),
+        possible_ground_truths=["1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["answer"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            'Voici une phrase en anglais contenant le pronom "it" dans un sens ambigu et sa traduction en français.'
+        )
+        .add_data(f"Phrase (originale en anglais) : {line['sentence']}")
+        .add_data(
+            f"Traduction en français (le pronom est caché par '_' ) : {line['context_fr']}"
+        )
+        .add_data("À quoi renvoie ce pronom ? Voici les choix: ")
+        .add_data(f"1 : {line['option1_fr']}")
+        .add_data(f"2 : {line['option2_fr']}")
+        .add_end("Réponds uniquement par 1 ou 2. La réponse est :")
+        .build(),
+        line_to_data_fn=lambda line: {
+            "sentence": line["sentence"],
+            "translation": line["context_fr"],
+            "referent1": line["option1_fr"],
+            "referent2": line["option2_fr"],
+        },
+    ),
+    COLETasks.WINO_X_MT.value: Dataset(
+        name="wino_x_mt",
+        description=(
+            "Pronoun resolution based on translations: choose between two French translations of an English "
+            "sentence with an ambiguous pronoun. The goal is to identify which of the two translations uses "
+            "the correct pronoun (he or she) based on the correct referent."
+        ),
+        possible_ground_truths=["1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["answer"]),
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Voici deux traductions d’une phrase anglaise contenant un pronom ambigu :"
+        )
+        .add_data(f"Phrase originale : {line['sentence']}")
+        .add_data(f"Traduction 1 (avec '{line['pronoun1']}') : {line['translation1']}")
+        .add_data(f"Traduction 2 (avec '{line['pronoun2']}') : {line['translation2']}")
+        .add_end(
+            "Quelle traduction utilise le bon pronom en fonction du référent visé dans la phrase originale ?\n"
+            "Réponds uniquement par 1 si la traduction 1 est correcte, ou 2 si la traduction 2 est correcte.\n"
+            "La réponse est :"
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "sentence": line["sentence"],
+            "translation1": line["translation1"],
+            "translation2": line["translation2"],
+            "pronoun1": line["pronoun1"],
+            "pronoun2": line["pronoun2"],
+        },
+    ),
+    COLETasks.MULTIBLIMP.value: Dataset(
+        name=COLETasks.MULTIBLIMP.value,
+        description="Choice task between two sentences : Choose the one which is grammatically correct.",
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(
+            line["label"]
+        ),  # The label is return as a string.
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise("Laquelle de ces phrases est grammaticalement correcte ?")
+            .add_data(f"Phrase 0:{line['sentence_a']}")
+            .add_data(f"Phrase 1:{line['sentence_b']}")
+            .add_end(
+                "Réponds avec seulement 0 si la phrase 0 "
+                "est grammaticalement correcte, et uniquement 1 si la phrase 1 est grammaticalement "
+                "correcte. La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {line["sentence_a"], line["sentence_b"]},
+    ),
+    COLETasks.FRACAS.value: Dataset(
+        name=COLETasks.FRACAS.value,
+        description="Natural language inference task : "
+        "predict the relation between two sentences (implication, neutral, contradiction).",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Quelle est la relation de la deuxième phrase par rapport à la première ?"
+        )
+        .add_data(line["premise"])
+        .add_data(line["hypothesis"])
+        .add_end(
+            (
+                "Réponds uniquement par :\n"
+                "0 - si la deuxième phrase implique la première,\n"
+                "1 - si la relation est neutre,\n"
+                "2 - s'il y a contradiction.\n"
+                "Réponds uniquement par 0, 1 ou 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "premise": line["premise"],
+            "hypothesis": line["hypothesis"],
+        },
+    ),
+    COLETasks.MMS.value: Dataset(
+        name=COLETasks.MMS.value,
+        description="A sentiment analysis task for classifying text as positive (2), negative (0), or neutral (1).",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise("Quel est le sentiment de cette phrase?")
+        .add_data(line["text"])
+        .add_end(
+            (
+                "Réponds uniquement par :\n"
+                "0 - si la phrase est négative,\n"
+                "1 - si la phrase est neutre,\n"
+                "2 - si la phrase est positive.\n"
+                "Réponds uniquement par 0, 1 ou 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "text": line["text"],
+        },
+    ),
+    COLETasks.WSD.value: Dataset(
+        name=COLETasks.WSD.value,
+        description="Extractive word sense disambiguation : Extract an ambiguous word in a sentence.",
+        possible_ground_truths=[],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Tu vas recevoir une phrase contenant un mot ambigu ainsi que les étiquettes du 'part-of-speech tagging "
+            "(PoS)' pour chaque mot de la phrase. Le mot ambigu peut être un verbe ou un adjectif.\n"
+            "Ta tâche est d’indiquer **exactement** ce mot ambigu dans la phrase, sans rien ajouter ni reformuler.\n"
+            "Réponds uniquement avec le mot ambigu identifié."
+        )
+        .add_data(f"Phrase : {line['sentence']}")
+        .add_data(f"Part-of-speech tagging: {line['pos_tag_labels']}")
+        .add_end("La réponse est :")
+        .build(),
+        line_to_data_fn=lambda line: {
+            "sentence": line["sentence"],
+            "pos_tag_labels": line["pos_tag_labels"],
+        },
+    ),
+    COLETasks.LINGNLI.value: Dataset(
+        name=COLETasks.LINGNLI.value,
+        description="Natural language inference task : "
+        "predict the relation between two sentences (implication, neutral, contradiction).",
+        possible_ground_truths=["0", "1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["label"],
+        line_to_prompt_fn=lambda line: PromptBuilder()
+        .add_premise(
+            "Quelle est la relation de la deuxième phrase par rapport à la première ?"
+        )
+        .add_data(line["premise"])
+        .add_data(line["hypothesis"])
+        .add_end(
+            (
+                "Réponds uniquement par :\n"
+                "0 - si la deuxième phrase implique la première,\n"
+                "1 - si la relation est neutre,\n"
+                "2 - s'il y a contradiction.\n"
+                "Réponds uniquement par 0, 1 ou 2. La réponse est :"
+            )
+        )
+        .build(),
+        line_to_data_fn=lambda line: {
+            "premise": line["premise"],
+            "hypothesis": line["hypothesis"],
+        },
+    ),
+    BorealTasks.TIMELINE.value: Dataset(
+        name=BorealTasks.TIMELINE.value,
+        description="Binary temporal ordering task: predict which of two events happened first.",
+        possible_ground_truths=["1", "2"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["answer"]),
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise(
+                "Deux événements historiques sont donnés. Lequel est survenu en premier ?"
+            )
+            .add_data(f"Événement 1 : {line['event_1']} (date: {line['date_1']})")
+            .add_data(f"Événement 2 : {line['event_2']} (date: {line['date_2']})")
+            .add_end(
+                "Réponds uniquement par 1 si l'événement 1 est venu avant l'événement 2, "
+                "ou 2 si l'événement 2 est venu avant l'événement 1. La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "event_1": line["event_1"],
+            "date_1": line["date_1"],
+            "event_2": line["event_2"],
+            "date_2": line["date_2"],
+        },
+    ),
+    BorealTasks.LQLE.value: Dataset(
+        name=BorealTasks.LQLE.value,
+        description=(
+            "Author name prediction for Quebec literature works: "
+            "given the title of a literary work, predict the Quebecois author name."
+        ),
+        possible_ground_truths=[],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["author"],
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise(
+                "On te donne le titre d'une oeuvre de littérature québécoise. "
+                "Ton rôle est de donner UNIQUEMENT le nom de l'auteur ou de l'autrice québécois(e) qui l'a écrit."
+            )
+            .add_data(f"Titre : {line.get('work_title', '').strip()}")
+            .add_end(
+                "Réponds uniquement par le nom complet de l'auteur ou de l'autrice, sans commentaire ni guillemets. "
+                "La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "work_title": line["work_title"],
+        },
+    ),
+    BorealTasks.PIQAQFR.value: Dataset(
+        name=BorealTasks.PIQAQFR.value,
+        description=(
+            "Physical commonsense multiple-choice task (Global PIQA) in Quebec French: "
+            "given a short situation and two possible solutions, choose the physically plausible one."
+        ),
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["label"]),
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise(
+                "Lis la situation suivante et choisis l'action qui a le plus de sens sur le plan physique."
+            )
+            .add_data(f"Problème : {line['prompt']}")
+            .add_data(f"Option 0 : {line['solution0']}")
+            .add_data(f"Option 1 : {line['solution1']}")
+            .add_end(
+                "Réponds uniquement par 0 si l'option 0 est la meilleure, "
+                "ou par 1 si l'option 1 est la meilleure. La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "prompt": line["prompt"],
+            "solution0": line["solution0"],
+            "solution1": line["solution1"],
+        },
+    ),
+    BorealTasks.PIQAFR.value: Dataset(
+        name=BorealTasks.PIQAFR.value,
+        description=(
+            "Physical commonsense multiple-choice task (Global PIQA) in standard French: "
+            "given a short situation and two possible solutions, choose the physically plausible one."
+        ),
+        possible_ground_truths=["0", "1"],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: str(line["label"]),
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise(
+                "Lis la situation suivante et choisis l'action qui a le plus de sens sur le plan physique."
+            )
+            .add_data(f"Problème : {line['prompt']}")
+            .add_data(f"Option 0 : {line['solution0']}")
+            .add_data(f"Option 1 : {line['solution1']}")
+            .add_end(
+                "Réponds uniquement par 0 si l'option 0 est la meilleure, "
+                "ou par 1 si l'option 1 est la meilleure. La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "prompt": line["prompt"],
+            "solution0": line["solution0"],
+            "solution1": line["solution1"],
+        },
+    ),
+    BorealTasks.QCCR.value: Dataset(
+        name=BorealTasks.QCCR.value,
+        description="Québec cities – predict the administrative region of a given city.",
+        possible_ground_truths=[],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["region"],
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise(
+                "À quelle région administrative appartient cette ville du Québec ?"
+            )
+            .add_data(f"Ville : {line['city']}")
+            .add_end(
+                "Réponds uniquement par le nom exact de la région. La réponse est :"
+            )
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "city": line["city"],
+        },
+    ),
+    BorealTasks.QCCY.value: Dataset(
+        name=BorealTasks.QCCY.value,
+        description="Québec cities – predict the year the city was founded.",
+        possible_ground_truths=[],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["founded_year"],
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise("En quelle année cette ville du Québec a-t-elle été fondée ?")
+            .add_data(f"Ville : {line['city']}")
+            .add_end("Réponds uniquement par l'année (4 chiffres). La réponse est :")
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "city": line["city"],
+        },
+    ),
+    BorealTasks.QCCP.value: Dataset(
+        name=BorealTasks.QCCP.value,
+        description="Québec cities – predict the population of the city (estimations de population, année 2024)",
+        possible_ground_truths=[],
+        hugging_face_repo=COLE_REPOSITORY_NAME,
+        line_to_truth_fn=lambda line: line["population"],
+        line_to_prompt_fn=lambda line: (
+            PromptBuilder()
+            .add_premise("Quelle est la population de cette ville du Québec en 2024 ?")
+            .add_data(f"Ville : {line['city']}")
+            .add_end("Réponds uniquement par le nombre d'habitants. La réponse est :")
+            .build()
+        ),
+        line_to_data_fn=lambda line: {
+            "city": line["city"],
+        },
+    ),
+}
+
+
+def preload_all_datasets():
+    """Loads all datasets into cache for later usage"""
+    for dataset in datasets.values():
+        dataset.load_data()
+
+
+def generate_metadata_dict():
+    """Generates a dictionary with all the datasets metadata information"""
+    metadata_dict = {}
+    for dataset in datasets.values():
+        metadata_dict[dataset.name] = dataset.metadata
+    return metadata_dict

cole/dataset/prompt_builder.py

@@ -0,0 +1,43 @@
+import logging
+from typing import List
+
+
+class PromptBuilder:
+    """Builder class for creating prompt strings with dynamic data."""
+
+    def __init__(self):
+        self.premise: List[str] = []
+        self.end: List[str] = []
+        self.data: List[str] = []
+        self.data_only = False
+
+    def add_data(self, data):
+        self.data.append(data)
+        return self
+
+    def add_end(self, end):
+        self.end.append(end)
+        return self
+
+    def set_data_only(self, data_only):
+        self.data_only = data_only
+        return self
+
+    def add_premise(self, premise):
+        self.premise.append(premise)
+        return self
+
+    def build(self):
+        """Builds and returns the prompt as a string based on data, premise and end that were added to the builder."""
+        if len(self.data) == 0:
+            logging.warning(
+                "This prompt did not contain any data, was that intentional ?"
+            )
+
+        data = "\n".join(self.data)
+        if self.data_only:
+            return data
+
+        end = "".join(self.end)
+        premise = "".join(self.premise)
+        return f"{premise}\n{data}\n{end}"

cole/docker_requirements.txt

@@ -0,0 +1,18 @@
+# Core
+python-dotenv
+numpy
+python-multipart
+scikit-learn
+# Web backend (si tu utilises FastAPI/Flask, sinon à ignorer)
+fastapi
+uvicorn
+slowapi
+# Optional: pretty printing, progress bars, etc.
+tqdm
+aenum
+
+evaluate
+wheel
+
+# Pour compatibilité ancienne
+protobuf<=7.36.0

cole/evaluation/evaluation_pipeline.py

@@ -0,0 +1,159 @@
+import argparse
+import gc
+import logging
+from datetime import datetime
+
+import torch
+import wandb
+from tqdm import tqdm
+
+from predictions.all_llms import llms
+from cole.evaluation.llm_evaluator import ModelEvaluator
+from cole.evaluation.llm_factory import model_factory
+from cole.evaluation.tools import split_llm_list, str2bool
+from cole.task.task_factory import tasks_factory
+from cole.task.task_names import COLETasks, BorealTasks
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    "--test",
+    help="If set to true, the system will default to testing only a small model with a few examples.",
+    default=False,
+    type=str2bool,
+)
+parser.add_argument(
+    "--max_examples",
+    "-m",
+    help="The maximum number of examples to use, defaults to None.",
+    type=int,
+    default=None,
+)
+parser.add_argument(
+    "--models_name",
+    "-mn",
+    help="The name of the model(s) to load.",
+    type=str,
+    default=None,
+)
+
+parser.add_argument(
+    "--batch_size",
+    help="The batch size to use during the evaluation.",
+    type=int,
+    default=32,
+)
+
+parser.add_argument(
+    "--llm_split",
+    help="The split of the LLMs list to use. It can be '1', '2' or '3'.",
+    type=int,
+    default=None,
+    choices=[1, 2, 3],
+)
+
+parser.add_argument(
+    "--skip_first_n",
+    help="The number of LLM to skip in the list of split",
+    type=int,
+    default=None,
+)
+parser.add_argument(
+    "--tasks_group",
+    help="The task group to test",
+    type=str,
+    default=None,
+    choices=["all", "cole", "boreal", "comparison"],
+)
+
+args = parser.parse_args()
+
+if args.tasks_group == "all":
+    tasks_names = list(COLETasks) + list(BorealTasks)
+    from cole import complete as project
+elif args.tasks_group == "cole":
+    tasks_names = list(COLETasks)
+    from cole import cole as project
+elif args.tasks_group == "boreal":
+    tasks_names = list(BorealTasks)
+    from cole import boreal as project
+elif args.tasks_group == "comparison":
+    tasks_names = ["frcoe"]
+    from cole import comparison as project
+else:
+    raise ValueError("Invalid value for tasks_group")
+
+tasks = tasks_factory(tasks_names)
+
+models = []
+if args.models_name is not None:
+    if args.models_name in llms:
+        models = llms[args.models_name]
+    else:
+        models = args.models_name.split(",")
+else:
+    models = llms["all"]
+
+models = split_llm_list(models=models, llm_split=args.llm_split)
+
+if args.skip_first_n is not None:
+    models = models[args.skip_first_n :]
+
+logging.info("Starting Evaluation")
+
+time_start = datetime.now()
+
+for model_name in tqdm(
+    models, total=len(models), desc="Processing LLM inference on tasks."
+):
+    model = None
+    evaluator = None
+    try:
+        model = model_factory(model_name, batch_size=args.batch_size)
+        logging.info("Creating model")
+        evaluator = ModelEvaluator()
+        logging.info("Evaluating model")
+
+        exp_name = f"{model_name}"
+        wandb.init(
+            project=project,
+            entity="doctorate",
+            config={
+                "model_name": model_name,
+                "tasks": "; ".join(
+                    t.value if hasattr(t, "value") else str(t) for t in tasks_names
+                ),
+                "batch_size": args.batch_size,
+            },
+            name=exp_name,
+        )
+
+        predictions_payload = evaluator.evaluate_subset(model, tasks, args.max_examples)
+        wandb.log(predictions_payload)
+
+        logging.info("Saving results")
+        evaluator.save_results("./results")
+
+        metrics_payload = evaluator.compute_metrics()
+        evaluator.save_metrics("./results")
+        wandb.log(metrics_payload)
+
+    except Exception as e:
+        error_message = f"Evaluation failed for model {model_name}: {e}"
+        logging.error(error_message)
+        wandb.finish(exit_code=1)
+        continue
+    else:
+        wandb.finish(exit_code=0)
+    finally:
+        # Memory cleaning
+        del model
+        del evaluator
+        gc.collect()
+        torch.cuda.empty_cache()
+
+time_end = datetime.now()
+info_message = f"End time: {time_end}"
+logging.info(info_message)
+elapsed_time = time_end - time_start
+info_message = f"Elapsed time: {elapsed_time}"
+logging.info(info_message)

cole/evaluation/evaluation_pipeline_private_llm.py

@@ -0,0 +1,137 @@
+import argparse
+import gc
+import logging
+from datetime import datetime
+
+import wandb
+from dotenv import load_dotenv
+from tqdm import tqdm
+
+from predictions.all_llms import private_llm
+from cole.evaluation.llm_evaluator import ModelEvaluator
+from cole.evaluation.tools import str2bool
+from cole.language_model.private_lm import RemoteLLMModel
+from cole.task.task_factory import tasks_factory
+from cole.task.task_names import COLETasks, BorealTasks
+
+load_dotenv(".env")
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    "--test",
+    help="If set to true, the system will default to testing only a small model with a few examples.",
+    default=False,
+    type=str2bool,
+)
+parser.add_argument(
+    "--max_examples",
+    "-m",
+    help="The maximum number of examples to use, defaults to None.",
+    type=int,
+    default=None,
+)
+parser.add_argument(
+    "--models_name",
+    "-mn",
+    help="The name of the model(s) to load.",
+    type=str,
+    default=None,
+)
+parser.add_argument(
+    "--provider_name",
+    "-pn",
+    help="The name of the LLM provider to load.",
+    type=str,
+    default=None,
+    choices=list(private_llm.keys()),
+)
+parser.add_argument(
+    "--tasks_group",
+    help="The task group to test",
+    type=str,
+    default=None,
+    choices=["all", "cole", "boreal", "comparison"],
+)
+
+args = parser.parse_args()
+
+if args.tasks_group == "all":
+    tasks_names = list(COLETasks) + list(BorealTasks)
+    from cole import complete as project
+elif args.tasks_group == "cole":
+    tasks_names = list(COLETasks)
+    from cole import cole as project
+elif args.tasks_group == "boreal":
+    tasks_names = list(BorealTasks)
+    from cole import boreal as project
+elif args.tasks_group == "comparison":
+    tasks_names = ["frcoe"]
+    from cole import comparison as project
+else:
+    raise ValueError("Invalid value for tasks_group")
+
+tasks = tasks_factory(tasks_names)
+
+models = []
+if args.models_name is not None:
+    if args.models_name in private_llm:
+        models = private_llm[args.models_name]
+    else:
+        models = args.models_name.split(",")
+elif args.provider_name is not None:
+    models = private_llm[args.provider_name]
+else:
+    models = private_llm["all"]
+
+logging.info("Starting Evaluation")
+
+time_start = datetime.now()
+
+for model_name in tqdm(
+    models, total=len(models), desc="Processing LLM inference on tasks."
+):
+    model = None
+    evaluator = None
+    try:
+        model = RemoteLLMModel(model_name=model_name)
+        logging.info("Creating model")
+        evaluator = ModelEvaluator()
+        logging.info("Evaluating model")
+
+        exp_name = f"{model_name}"
+        wandb.init(
+            project=project,
+            entity="doctorate",
+            config={"model_name": model_name, "tasks": "; ".join(tasks_names)},
+            name=exp_name,
+        )
+
+        predictions_payload = evaluator.evaluate_subset(model, tasks, args.max_examples)
+        logging.info("Writing predictions to WandB.")
+        wandb.log(predictions_payload)
+
+        logging.info("Saving results")
+        evaluator.save_results("./results")
+
+        metrics_payload = evaluator.compute_metrics()
+        evaluator.save_metrics("./results")
+        wandb.log(metrics_payload)
+
+    except Exception as e:
+        error_message = f"Evaluation failed for model {model_name}: {e}"
+        logging.error(error_message)
+        wandb.finish(exit_code=1)
+        continue
+    finally:
+        # Memory cleaning
+        del model
+        del evaluator
+        gc.collect()
+        wandb.finish(exit_code=0)
+
+time_end = datetime.now()
+info_message = f"End time: {time_end}"
+logging.info(info_message)
+elapsed_time = time_end - time_start
+info_message = f"Elapsed time: {elapsed_time}"
+logging.info(info_message)

cole/evaluation/evaluation_pipeline_small.py

@@ -0,0 +1,150 @@
+import argparse
+import gc
+import logging
+from datetime import datetime
+
+import torch
+import wandb
+from tqdm import tqdm
+
+from predictions.all_llms import small_llm
+from cole.evaluation.llm_evaluator import ModelEvaluator
+from cole.evaluation.llm_factory import model_factory
+from cole.evaluation.tools import str2bool
+from cole.task.task_factory import tasks_factory
+from cole.task.task_names import COLETasks, BorealTasks
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    "--test",
+    help="If set to true, the system will default to testing only a small model with a few examples.",
+    default=False,
+    type=str2bool,
+)
+parser.add_argument(
+    "--max_examples",
+    "-m",
+    help="The maximum number of examples to use, defaults to None.",
+    type=int,
+    default=None,
+)
+parser.add_argument(
+    "--token",
+    "-t",
+    help="Input your HuggingFace token to fetch models.",
+    type=str,
+    default=None,
+)
+parser.add_argument(
+    "--models_name",
+    "-mn",
+    help="The name of the model(s) to load.",
+    type=str,
+    default=None,
+)
+
+parser.add_argument(
+    "--batch_size",
+    help="The batch size to use during the evaluation.",
+    type=int,
+    default=32,
+)
+
+parser.add_argument(
+    "--skip_first_n",
+    help="The number of LLM to skip in the list of split",
+    type=int,
+    default=None,
+)
+
+parser.add_argument(
+    "--tasks_group",
+    help="The task group to test",
+    type=str,
+    default=None,
+    choices=["all", "cole", "boreal", "comparison"],
+)
+
+args = parser.parse_args()
+
+if args.tasks_group == "all":
+    tasks_names = list(COLETasks) + list(BorealTasks)
+    from cole import complete as project
+elif args.tasks_group == "cole":
+    tasks_names = list(COLETasks)
+    from cole import cole as project
+elif args.tasks_group == "boreal":
+    tasks_names = list(BorealTasks)
+    from cole import boreal as project
+elif args.tasks_group == "comparison":
+    tasks_names = ["frcoe"]
+    from cole import comparison as project
+else:
+    raise ValueError("Invalid value for tasks_group")
+
+tasks = tasks_factory(tasks_names)
+
+models = []
+if args.models_name is not None:
+    if args.models_name in small_llm:
+        models = small_llm[args.models_name]
+    else:
+        models = args.models_name.split(",")
+else:
+    models = small_llm["all"]
+
+if args.skip_first_n is not None:
+    models = models[args.skip_first_n :]
+
+logging.info("Starting Evaluation")
+
+time_start = datetime.now()
+
+for model_name in tqdm(
+    models, total=len(models), desc="Processing LLM inference on tasks."
+):
+    model = None
+    evaluator = None
+    try:
+        model = model_factory(model_name, batch_size=args.batch_size)
+        logging.info("Creating model")
+        evaluator = ModelEvaluator()
+        logging.info("Evaluating model")
+
+        exp_name = f"{model_name}"
+        wandb.init(
+            project=project,
+            entity="doctorate",
+            config={"model_name": model_name, "tasks": "; ".join(tasks_names)},
+            name=exp_name,
+        )
+
+        predictions_payload = evaluator.evaluate_subset(model, tasks, args.max_examples)
+        wandb.log(predictions_payload)
+
+        logging.info("Saving results")
+        evaluator.save_results("./results")
+
+        metrics_payload = evaluator.compute_metrics()
+        evaluator.save_metrics("./results")
+        wandb.log(metrics_payload)
+
+    except Exception as e:
+        error_message = f"Evaluation failed for model {model_name}: {e}"
+        logging.error(error_message)
+        wandb.finish(exit_code=1)
+        continue
+    finally:
+        # Memory cleaning
+        del model
+        del evaluator
+        gc.collect()
+        torch.cuda.empty_cache()
+        wandb.finish(exit_code=0)
+
+time_end = datetime.now()
+info_message = f"End time: {time_end}"
+logging.info(info_message)
+elapsed_time = time_end - time_start
+info_message = f"Elapsed time: {elapsed_time}"
+logging.info(info_message)

cole/evaluation/evaluation_pipeline_small_2.py

@@ -0,0 +1,151 @@
+import argparse
+import gc
+import logging
+from datetime import datetime
+
+import torch
+import wandb
+from tqdm import tqdm
+
+from predictions.all_llms import small_llm_2
+from cole.evaluation.llm_evaluator import ModelEvaluator
+from cole.evaluation.llm_factory import model_factory
+from cole.evaluation.tools import str2bool
+from cole.task.task_factory import tasks_factory
+from cole.task.task_names import COLETasks, BorealTasks
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    "--test",
+    help="If set to true, the system will default to testing only a small model with a few examples.",
+    default=False,
+    type=str2bool,
+)
+parser.add_argument(
+    "--max_examples",
+    "-m",
+    help="The maximum number of examples to use, defaults to None.",
+    type=int,
+    default=None,
+)
+parser.add_argument(
+    "--token",
+    "-t",
+    help="Input your HuggingFace token to fetch models.",
+    type=str,
+    default=None,
+)
+parser.add_argument(
+    "--models_name",
+    "-mn",
+    help="The name of the model(s) to load.",
+    type=str,
+    default=None,
+)
+
+parser.add_argument(
+    "--batch_size",
+    help="The batch size to use during the evaluation.",
+    type=int,
+    default=256,
+)
+
+parser.add_argument(
+    "--skip_first_n",
+    help="The number of LLM to skip in the list of split",
+    type=int,
+    default=None,
+)
+parser.add_argument(
+    "--tasks_group",
+    help="The task group to test",
+    type=str,
+    default=None,
+    choices=["all", "cole", "boreal", "comparison"],
+)
+
+args = parser.parse_args()
+
+if args.tasks_group == "all":
+    tasks_names = list(COLETasks) + list(BorealTasks)
+    from cole import complete as project
+elif args.tasks_group == "cole":
+    tasks_names = list(COLETasks)
+    from cole import cole as project
+elif args.tasks_group == "boreal":
+    tasks_names = list(BorealTasks)
+    from cole import boreal as project
+elif args.tasks_group == "comparison":
+    tasks_names = ["frcoe"]
+    from cole import comparison as project
+else:
+    raise ValueError("Invalid value for tasks_group")
+
+tasks = tasks_factory(tasks_names)
+
+models = []
+if args.models_name is not None:
+    if args.models_name in small_llm_2:
+        models = small_llm_2[args.models_name]
+    elif args.models_name == "RandomBaselineModel":
+        models = ["RandomBaselineModel"]
+    else:
+        models = args.models_name.split(",")
+else:
+    models = small_llm_2["all"]
+
+if args.skip_first_n is not None:
+    models = models[args.skip_first_n :]
+
+logging.info("Starting Evaluation")
+
+time_start = datetime.now()
+
+for model_name in tqdm(
+    models, total=len(models), desc="Processing LLM inference on tasks."
+):
+    model = None
+    evaluator = None
+    try:
+        model = model_factory(model_name, batch_size=args.batch_size)
+        logging.info("Creating model")
+        evaluator = ModelEvaluator()
+        logging.info("Evaluating model")
+
+        exp_name = f"{model_name}"
+        wandb.init(
+            project=project,
+            entity="doctorate",
+            config={"model_name": model_name, "tasks": "; ".join(tasks_names)},
+            name=exp_name,
+        )
+
+        predictions_payload = evaluator.evaluate_subset(model, tasks, args.max_examples)
+        wandb.log(predictions_payload)
+
+        logging.info("Saving results")
+        evaluator.save_results("./results")
+
+        metrics_payload = evaluator.compute_metrics()
+        evaluator.save_metrics("./results")
+        wandb.log(metrics_payload)
+
+    except Exception as e:
+        error_message = f"Evaluation failed for model {model_name}: {e}"
+        logging.error(error_message)
+        wandb.finish(exit_code=1)
+        continue
+    finally:
+        # Memory cleaning
+        del model
+        del evaluator
+        gc.collect()
+        torch.cuda.empty_cache()
+        wandb.finish(exit_code=0)
+
+time_end = datetime.now()
+info_message = f"End time: {time_end}"
+logging.info(info_message)
+elapsed_time = time_end - time_start
+info_message = f"Elapsed time: {elapsed_time}"
+logging.info(info_message)

cole/evaluation/llm_evaluator.py

@@ -0,0 +1,217 @@
+import gc
+import json
+import logging
+import os
+from datetime import datetime
+from typing import Dict, List
+
+import torch
+import wandb
+from datasets import Dataset
+from tqdm import tqdm
+
+from cole.language_model.language_model_abstraction import LanguageModel
+from cole.task.task import Task
+from cole.task.task_factory import tasks_factory
+
+
+class ModelEvaluator:
+    """
+    The model evaluator acts as a pipeline for evaluating models on tasks available from tasks_factory.
+    """
+
+    def __init__(self):
+        self.last_predictions = {}
+
+        self.last_metrics = {}
+
+        self.last_model_name = None
+
+    def compute_metrics(self) -> Dict:
+        """
+        Compute metrics over the last tested model's predictions,
+        must have called one the evaluate functions before or loaded predictions with load_predictions_from_file.
+        """
+        task_metrics: List[Dict] = []
+
+        for task_dict in tqdm(
+            self.last_predictions["tasks"], desc="Computing metrics: "
+        ):
+            task_name, preds = list(task_dict.items())[0]
+            if not preds:
+                warning_message = f"Task '{task_name}' ignored due to no predictions"
+                logging.warning(warning_message)
+                continue
+            try:
+                tasks = tasks_factory([task_name])
+                task = tasks[0]
+                metric_score, warning = task.compute(preds)
+            except Exception as e:
+                error_message = f"Error while calculating metrics'{task_name}' : {e}"
+                logging.error(error_message)
+                continue
+            metric_name = task.metric_name
+            task_entry = {
+                task_name: {
+                    metric_name: {**metric_score, f"{metric_name}_warning": warning}
+                }
+            }
+            task_metrics.append(task_entry)
+            wandb.log({f"{task_name}.{metric_name}": {**metric_score}})
+
+        # Build the response from a fresh dict so `last_metrics` does NOT alias
+        # `last_predictions`. The previous implementation stored the same dict
+        # reference in both, so mutating one would silently mutate the other.
+        self.last_metrics = {
+            **{k: v for k, v in self.last_predictions.items() if k != "tasks"},
+            "tasks": task_metrics,
+        }
+        return self.last_metrics
+
+    def load_predictions_from_file(self, file_path: str) -> None:
+        """
+        Load predictions from file to compute metrics :param file_path:path to the predictions file.
+        """
+
+        try:
+            with open(file_path, "r", encoding="utf-8") as f:
+                self.last_predictions = json.load(f)
+        except FileNotFoundError:
+            error = f"File not found: {file_path}"
+            logging.error(error)
+            self.last_predictions = None
+        except json.JSONDecodeError:
+            error = f"Invalid JSON in file: {file_path}"
+            logging.error(error)
+            self.last_predictions = None
+
+    def save_metrics(self, save_path):
+        """
+        Saves computed metrics to a json file.
+        :param save_path : the path to which the json file will be saved.
+        """
+        # `last_metrics` is initialized to {} (not None), so the previous
+        # `is None` check never fired. Treat both empty/missing the same way.
+        if not self.last_metrics or self.last_model_name is None:
+            logging.info("No metrics saved")
+            return None
+        return self.save_object(
+            save_path,
+            self.last_metrics,
+            f"{self.last_model_name.replace('/', '_')}_metrics.json",
+        )
+
+    def evaluate(self, model: LanguageModel, tasks: List[Task]):
+        """
+        Evaluates a given model on the given tasks.
+        :param model : the model that will infer on the given tasks.
+        :param tasks : the tasks to be evaluated on.
+        """
+        return self.evaluate_subset(model, tasks)
+
+    def evaluate_subset(
+        self, model: LanguageModel, tasks: List[Task], subset_size=None
+    ) -> Dict:
+        """
+        Evaluates a given model on the given tasks, but only on a given size.
+        :param model : the model that will infer on the given tasks.
+        :param tasks : the tasks to be evaluated on.
+        :param subset_size : the size of the subset to be evaluated.
+        """
+        predictions = []
+        for task in tasks:
+            info_log = (
+                f"-----Doing task '{task.task_name}' with model '{model.name}-----'."
+            )
+            logging.info(info_log)
+            # Initialize before try so the finally block can drop them without
+            # relying on `locals()` introspection (which is brittle and was
+            # papering over the case where the try block raised before assignment).
+            prompts = None
+            evaluate_dataset = None
+            try:
+                if subset_size is None:
+                    prompts = task.dataset.prompts[:]
+                else:
+                    prompts = task.dataset.prompts[:subset_size]
+
+                evaluate_dataset = Dataset.from_dict({"text": prompts})
+
+                task_predictions = model.predict(
+                    evaluation_dataset=evaluate_dataset, task=task
+                )
+
+                task_predictions = {task.task_name: task_predictions}
+                predictions.append(task_predictions)
+
+            except Exception as e:
+                error_message = f"Task '{task.task_name}' has failed : {e}"
+                logging.error(error_message)
+                wandb.log({task.task_name: "Failed"})
+                continue
+            finally:
+                # Memory cleaning
+                del evaluate_dataset
+                del prompts
+                torch.cuda.empty_cache()
+                gc.collect()
+            wandb.log({task.task_name: "Success"})
+        logging.info("Finished evaluating tasks.")
+        self.last_predictions = {
+            "model_name": model.name,
+            "model_url": f"https://huggingface.co/{model.name}",
+            "tasks": predictions,
+        }
+        self.last_model_name = model.name
+        return self.last_predictions
+
+    def save_results(self, save_path):
+        """
+        Saves inferred metrics to a json file.
+        :param save_path : the path to which the json file will be saved.
+        """
+
+        if self.last_model_name is None:
+            logging.error("Please evaluate before saving results")
+            return None
+        date_time_stamp = datetime.now().strftime("%Y%m%d-%H%M")
+        return self.save_object(
+            save_path,
+            self.last_predictions,
+            f"{self.last_model_name.replace('/', '_')}_{date_time_stamp}.json",
+        )
+
+    def save_object(self, save_dir_path, saved_object, filename):
+        """
+        Utility method to save the given object into a json file.
+        """
+        os.makedirs(save_dir_path, exist_ok=True)
+        full_path = os.path.join(save_dir_path, filename)
+        if os.path.isfile(full_path):
+            logging.info("Appending results to previous results file.")
+            try:
+                with open(full_path, "r", encoding="utf-8") as f:
+                    data = json.load(f)
+                # `data.get("tasks")` could be None on a malformed/legacy file,
+                # in which case `.extend(...)` would raise AttributeError.
+                # Likewise the new payload may have no "tasks" key.
+                existing_tasks = data.get("tasks") or []
+                new_tasks = saved_object.get("tasks") or []
+                data["tasks"] = existing_tasks + new_tasks
+                with open(full_path, "w", encoding="utf-8") as f:
+                    json.dump(data, f, indent=2)
+                info_message = f"Results saved to {save_dir_path}"
+                logging.info(info_message)
+            except Exception as e:
+                error_message = f"Failed to save object: {e}"
+                logging.error(error_message)
+        else:
+            try:
+                with open(full_path, "w", encoding="utf-8") as f:
+                    json.dump(saved_object, f, indent=2)
+                info_message = f"Results saved to {save_dir_path}"
+                logging.info(info_message)
+            except Exception as e:
+                error_message = f"Failed to save object: {e}"
+                logging.error(error_message)
+        return full_path

cole/evaluation/llm_factory.py

@@ -0,0 +1,25 @@
+from typing import Union
+
+from predictions.all_llms import small_llm, llms, private_llm, small_llm_2
+from cole.language_model.baseline import RandomBaselineModel
+from cole.language_model.hugging_face_lm import HFLLMModel
+from cole.language_model.language_model_abstraction import LanguageModel
+from cole.language_model.private_lm import RemoteLLMModel
+
+
+def model_factory(
+    model_name: str, batch_size: Union[int, None] = None
+) -> LanguageModel:
+    if model_name == "RandomBaselineModel":
+        model = RandomBaselineModel(model_name="random_baseline")
+    elif model_name in private_llm["all"]:
+        model = RemoteLLMModel(model_name=model_name)
+    elif (
+        model_name in llms["all"]
+        or model_name in small_llm["all"]
+        or model_name in small_llm_2["all"]
+    ):
+        model = HFLLMModel(model_name=model_name, batch_size=batch_size)
+    else:
+        raise ValueError(f"Model {model_name} not supported.")
+    return model

cole/evaluation/tools.py

@@ -0,0 +1,33 @@
+import argparse
+from typing import List, Union
+
+
+def split_llm_list(models: List, llm_split: Union[None, int]) -> List:
+    if llm_split is None:
+        return models
+    if llm_split not in (1, 2, 3):
+        raise ValueError("llm_split must be in [1, 2, 3].")
+    if llm_split == 1:
+        models = models[: len(models) // 3]
+    elif llm_split == 2:
+        models = models[len(models) // 3 : 2 * len(models) // 3]
+    elif llm_split == 3:
+        models = models[2 * len(models) // 3 :]
+    return models
+
+
+def str2bool(value: Union[bool, str]) -> bool:
+    """argparse-friendly bool parser.
+
+    `argparse(type=bool)` is a footgun: `bool("False")` is True, so any
+    non-empty string flips the flag to True. This converter rejects garbage
+    explicitly and accepts the obvious truthy/falsy spellings.
+    """
+    if isinstance(value, bool):
+        return value
+    normalized = str(value).strip().lower()
+    if normalized in ("true", "1", "yes", "y", "t"):
+        return True
+    if normalized in ("false", "0", "no", "n", "f"):
+        return False
+    raise argparse.ArgumentTypeError(f"Boolean value expected, got: {value!r}")

cole/language_model/anthropic_wrapper.py

@@ -0,0 +1,20 @@
+from typing import Dict
+
+from anthropic import Anthropic
+
+from cole.language_model.open_ai_api_lm_wrapper import OpenAIAPILMWrapper
+
+
+class AnthropicWrapper(OpenAIAPILMWrapper):
+    def __init__(self, model_name: str, api_key: str, extra_params: Dict):
+        super().__init__(model_name=model_name, extra_params=extra_params)
+        self.client = Anthropic(api_key=api_key)
+
+        self.tool_choices = {"type": "tool", "name": "classification"}
+
+    def _inner_generate_fn(self, prompt: Dict):
+        return self.client.messages.create(
+            model=self.model_name,
+            messages=prompt,
+            **self._extra_params,
+        )

cole/language_model/baseline.py

@@ -0,0 +1,45 @@
+import numpy as np
+from datasets.formatting.formatting import LazyRow
+
+from cole.language_model.language_model_abstraction import LanguageModel
+from cole.task.task import Task
+
+
+class RandomBaselineModel(LanguageModel):
+    def __init__(self, model_name: str, seed: int = 42):
+        super().__init__(model_name)
+        self.random_generator = np.random.RandomState(seed=seed)
+
+    def predict(self, evaluation_dataset, task: Task):
+        size = len(evaluation_dataset)
+        choices = task.dataset.possible_ground_truths
+        if len(choices) == 0:
+            # Meaning it is a generation task.
+            predictions = []
+            for row in evaluation_dataset:
+
+                # Since we work with the prompt, including instruction, we extract the instance sentence after the
+                # "Phrase :", then we remove the trailing Part-of-speech content.
+                instance_sentence = row["text"].split("Phrase : ")[-1].split("\n")[0]
+                whitespace_instance_sentence = instance_sentence.split(" ")
+
+                choices = range(0, len(whitespace_instance_sentence))
+                prediction_idx = self.random_generator.choice(choices, size=1).tolist()[
+                    0
+                ]
+                prediction = whitespace_instance_sentence[prediction_idx]
+                predictions.append(prediction)
+        else:
+            # Meaning it is an inference task.
+            predictions = self.random_generator.choice(choices, size=size).tolist()
+        return predictions
+
+    def infer(self, rows: LazyRow) -> LazyRow:
+        return rows
+
+    def generate(self, rows: LazyRow) -> LazyRow:
+        return rows
+
+    @property
+    def num_parameters(self):
+        return 0

cole/language_model/cohere_wrapper.py

@@ -0,0 +1,31 @@
+from typing import Dict, List, Union
+
+from openai import OpenAI
+
+from cole.language_model.open_ai_api_lm_wrapper import OpenAIAPILMWrapper
+
+
+class CohereWrapper(OpenAIAPILMWrapper):
+    def __init__(
+        self,
+        model_name: str,
+        api_key: str,
+        extra_params: Dict,
+        use_function_calling: bool,
+        base_url: Union[None, str] = None,
+        timeout: int = 480,
+    ):
+        super().__init__(
+            model_name=model_name,
+            extra_params=extra_params,
+            use_function_calling=use_function_calling,
+        )
+        self.client = OpenAI(api_key=api_key, base_url=base_url, timeout=timeout)
+
+    def _inner_generate_fn(self, prompt: List):
+        return self.client.chat.completions.create(
+            model=self.model_name,
+            messages=prompt,
+            stream=False,
+            **self._extra_params,
+        )

cole/language_model/deepseek_wrapper.py

@@ -0,0 +1,30 @@
+from typing import Dict, List
+
+from openai import OpenAI
+
+from cole.language_model.open_ai_api_lm_wrapper import OpenAIAPILMWrapper
+
+
+class DeepSeekWrapper(OpenAIAPILMWrapper):
+    def __init__(
+        self,
+        model_name: str,
+        api_key: str,
+        extra_params: Dict,
+        use_function_calling: bool,
+    ):
+        super().__init__(
+            model_name=model_name,
+            extra_params=extra_params,
+            use_function_calling=use_function_calling,
+        )
+        self.client = OpenAI(api_key=api_key, base_url="https://api.deepseek.com")
+        self.tool_choices = "auto"
+
+    def _inner_generate_fn(self, prompt: List):
+        return self.client.chat.completions.create(
+            model=self.model_name,
+            messages=prompt,
+            stream=False,
+            **self._extra_params,
+        )

cole/language_model/google_wrapper.py

@@ -0,0 +1,22 @@
+from typing import Dict, List
+
+from openai import OpenAI
+
+from cole.language_model.open_ai_api_lm_wrapper import OpenAIAPILMWrapper
+
+
+class GoogleWrapper(OpenAIAPILMWrapper):
+    def __init__(self, model_name: str, api_key: str, extra_params: Dict):
+        super().__init__(model_name=model_name, extra_params=extra_params)
+        self.client = OpenAI(
+            api_key=api_key,
+            base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
+        )
+
+    def _inner_generate_fn(self, prompt: List):
+        return self.client.chat.completions.create(
+            model=self.model_name,
+            messages=prompt,
+            stream=False,
+            **self._extra_params,
+        )

cole/language_model/hugging_face_lm.py

@@ -0,0 +1,161 @@
+from typing import Union, List, Dict
+
+import torch
+from datasets import Dataset
+from datasets.formatting.formatting import LazyRow
+from transformers import (
+    pipeline,
+)
+
+from cole.language_model.language_model_abstraction import LanguageModel
+from cole.language_model.huggingface_language_model_factory import (
+    hugging_face_language_model_tokenizer_factory,
+)
+from cole.task.task import TaskType, Task
+
+
+class HFLLMModel(LanguageModel):
+    """
+    LLM Model based on Hugging Face Transformers and pipeline mechanism, loads pretrained LLM models and uses
+    it for inference.
+    """
+
+    def __init__(
+        self,
+        model_name: str,
+        token: Union[str, None] = None,
+        batch_size: int = 8,
+    ):
+        super().__init__(model_name)
+        self._model_name = model_name
+        self._token = token
+
+        self.model, self.tokenizer = hugging_face_language_model_tokenizer_factory(
+            model_name=self._model_name,
+            huggingface_token=self._token,
+        )
+
+        num_params = self.model.num_parameters()
+
+        # To handle max batch size for these models.
+        if num_params >= 70000000000:  # 70B
+            batch_size = 2
+        elif num_params >= 32000000000:  # 32B
+            batch_size = 8
+        elif num_params >= 27000000000:  # 27B
+            batch_size = 16
+        elif "gpt-oss" in self._model_name:
+            batch_size = 8  # Otherwise a lot of OOM
+            self._batch_size_sts22 = (
+                1  # For sts22, GPT-oss get OOM for batch size higher than 1.
+            )
+        self._batch_size = batch_size
+
+    def predict(self, evaluation_dataset: Dataset, task: Task) -> List:
+        if task.task_name == "sts22" and "gpt-oss" in self._model_name:
+            # For sts22, GPT-oss get OOM for batch size higher than 1.
+            batch_size = self._batch_size_sts22
+        else:
+            batch_size = self._batch_size
+
+        if task.task_type == TaskType.INFERENCE:
+            labels = task.dataset.possible_ground_truths
+            self.pipeline = pipeline(
+                task="zero-shot-classification",
+                model=self.model,
+                tokenizer=self.tokenizer,
+                batch_size=batch_size,
+                dtype="float16",
+                return_full_text=False,
+                max_new_tokens=16,
+                padding=True,
+                truncation=True,
+                max_length=4096,
+                candidate_labels=labels,
+            )
+            if len(labels) == 2:
+                inference_fn = self.infer_binary
+            else:
+                inference_fn = self.infer
+        else:
+            self.pipeline = pipeline(
+                task="text-generation",
+                model=self.model,
+                tokenizer=self.tokenizer,
+                batch_size=batch_size,
+                dtype="float16",
+                return_full_text=False,
+                max_new_tokens=64,
+                padding=True,
+                truncation=True,
+                max_length=4096,
+            )
+            inference_fn = self.generate
+
+        process_dataset = evaluation_dataset.map(
+            inference_fn,
+            batched=True,
+            batch_size=self._batch_size,
+            desc=f"Running evaluation for task: {task.task_name}",
+            remove_columns="text",
+        )
+
+        return list(process_dataset["prediction"])
+
+    def generate(self, rows: LazyRow) -> Dict:
+        """
+        Do a generation over a set of rows and extract the generated text and apply string post-processing.
+        """
+        with torch.no_grad():
+            text = rows["text"]
+
+            if self._model_name.lower() == "chocolatine":
+                # Problem with Phi-4 generation:
+                # https://github.com/huggingface/transformers/issues/36071#issuecomment-3109331152
+                generation_args = {"use_cache": False}
+                outputs = self.pipeline(text, **generation_args)
+            else:
+                outputs = self.pipeline(text)
+
+            generated_texts = [
+                output[0]["generated_text"].strip() for output in outputs
+            ]
+
+        return {"prediction": generated_texts}
+
+    def infer(self, rows: LazyRow) -> Dict:
+        """
+        Do a zero-shot classification and extract the label using a per-element generation.
+
+        For a fucking strange reason, the pipeline does not work in this case:
+        1. Batched generation of more than one element
+        2. More than 2 labels.
+
+        Thus, we need to loop over the element. Painful I know.
+        """
+
+        with torch.no_grad():
+            texts = rows["text"]
+
+            classifications = []
+            for text in texts:
+                output = self.pipeline(text)
+                classifications.append(
+                    output["labels"][0]
+                )  # Labels are sorted in likelihood order.
+
+        return {"prediction": classifications}
+
+    def infer_binary(self, rows: LazyRow) -> Dict:
+        """
+        Do a binary zero-shot classification and extract the label using a per-element generation.
+        """
+        with torch.no_grad():
+            texts = rows["text"]
+
+            outputs = self.pipeline(texts)
+            classifications = [
+                output["labels"][0] for output in outputs
+            ]  # Labels are sorted in likelihood order.
+
+        return {"prediction": classifications}

cole/language_model/huggingface_language_model_factory.py

@@ -0,0 +1,68 @@
+import torch
+from transformers import (
+    AutoTokenizer,
+    BitsAndBytesConfig,
+    AutoModelForCausalLM,
+)
+from unsloth import FastLanguageModel
+
+
+def hugging_face_language_model_tokenizer_factory(
+    model_name,
+    huggingface_token: str,
+):
+    if (
+        "chocolatine" in model_name.lower()
+        or "lucie" in model_name.lower()
+        or "mixtral" in model_name.lower()
+        or "eurollm" in model_name.lower()
+        or "ibm-granite" in model_name.lower()
+        or "swiss-ai" in model_name.lower()
+    ):
+
+        compute_dtype = getattr(torch, "bfloat16")
+        bnb_configs = BitsAndBytesConfig(
+            load_in_4bit=True,
+            bnb_4bit_quant_type="nf4",
+            bnb_4bit_compute_dtype=compute_dtype,
+            bnb_4bit_use_double_quant=True,
+        )
+        if "chocolatine" in model_name.lower() or "mixtral" in model_name.lower():
+            attn_implementation = "flash_attention_2"
+        else:
+            attn_implementation = "sdpa"
+        model = AutoModelForCausalLM.from_pretrained(
+            model_name,
+            token=huggingface_token,
+            quantization_config=bnb_configs,
+            load_in_8bit=False,  # Since we use 4bits
+            trust_remote_code=True,
+            attn_implementation=attn_implementation,
+            dtype=torch.float16,
+        )
+        if "chocolatine" in model_name.lower():
+            extra_args = {"padding_side": "left"}
+        else:
+            extra_args = {}
+
+        if "eurollm" in model_name.lower():
+            model.gradient_checkpointing_enable()
+
+        tokenizer = AutoTokenizer.from_pretrained(
+            model_name, token=huggingface_token, **extra_args
+        )
+
+        if tokenizer.pad_token is None:
+            tokenizer.pad_token_id = model.config.eos_token_id
+    else:
+        model, tokenizer = FastLanguageModel.from_pretrained(
+            model_name,
+            max_seq_length=4096,
+            device_map="sequential",
+            dtype=None,
+            load_in_4bit=True,
+            token=huggingface_token,
+        )
+
+    model.eval()
+    return model, tokenizer

cole/language_model/init_function_calling.py

@@ -0,0 +1,51 @@
+from typing import List
+
+
+def init_function_calling(labels: List[str], tool_choices: str, open_ai_api_call: bool):
+    if open_ai_api_call:
+        call = {
+            "tools": [
+                {
+                    "type": "function",
+                    "function": {
+                        "name": "classification",
+                        "description": "Use this function to return your response to the user question.",
+                        "parameters": {
+                            "type": "object",
+                            "properties": {
+                                "category": {
+                                    "type": "string",
+                                    "enum": labels,
+                                    "description": "The permitted categories to response to the question.",
+                                },
+                            },
+                            "required": ["category"],
+                        },
+                    },
+                }
+            ],
+            "tool_choice": tool_choices,
+        }
+    else:
+        # Anthropic call
+        call = {
+            "tools": [
+                {
+                    "name": "classification",
+                    "description": "Use this function to return your response to the user question.",
+                    "input_schema": {
+                        "type": "object",
+                        "properties": {
+                            "category": {
+                                "type": "string",
+                                "enum": labels,
+                                "description": "The permitted categories to response to the question.",
+                            },
+                        },
+                        "required": ["category"],
+                    },
+                },
+            ],
+            "tool_choice": tool_choices,
+        }
+    return call

cole/language_model/language_model_abstraction.py

@@ -0,0 +1,26 @@
+# pylint: disable=method-hidden
+
+from abc import abstractmethod, ABC
+from typing import List
+
+from datasets import Dataset
+from datasets.formatting.formatting import LazyRow
+
+from cole.task.task import Task
+
+
+class LanguageModel(ABC):
+    def __init__(self, model_name: str):
+        self.name = model_name
+
+    @abstractmethod
+    def predict(self, evaluation_dataset: Dataset, task: Task) -> List:
+        raise NotImplementedError
+
+    @abstractmethod
+    def infer(self, rows: LazyRow):
+        raise NotImplementedError
+
+    @abstractmethod
+    def generate(self, rows: LazyRow):
+        raise NotImplementedError

cole/language_model/mistral_wrapper.py

@@ -0,0 +1,20 @@
+from typing import Dict
+
+from mistralai import Mistral
+
+from cole.language_model.open_ai_api_lm_wrapper import OpenAIAPILMWrapper
+
+
+class MistralWrapper(OpenAIAPILMWrapper):
+    def __init__(self, model_name: str, api_key: str, extra_params: Dict):
+        super().__init__(model_name=model_name, extra_params=extra_params)
+        self.client = Mistral(api_key=str(api_key))
+        self.tool_choices = "any"
+
+    def _inner_generate_fn(self, prompt: Dict):
+        return self.client.chat.complete(
+            model=self.model_name,
+            messages=prompt,
+            n=1,
+            **self._extra_params,
+        )

cole/language_model/open_ai_api_lm_wrapper.py

@@ -0,0 +1,171 @@
+# pylint: disable=inconsistent-return-statements
+import logging
+import time
+from abc import ABC, abstractmethod
+from typing import List, Dict
+
+from cole import NA_VALUE
+from cole.language_model.init_function_calling import init_function_calling
+
+
+class OpenAIAPILMWrapper(ABC):
+    def __init__(
+        self,
+        model_name: str,
+        extra_params: Dict,
+        use_function_calling: bool = True,
+        max_retries: int = 10,
+    ):
+        self.model_name = model_name
+        self._extra_params = extra_params
+        self.max_retries = max_retries
+        self.use_function_calling = use_function_calling
+        self.tool_choices = "required"
+
+        self._none_prediction_counter = 0
+        self._max_retries_counter = 0
+
+    @abstractmethod
+    def _inner_generate_fn(self, prompt: List) -> Dict:
+        pass
+
+    def init_function_calling(self, labels: List[str], tool_choices: str) -> None:
+        if self.use_function_calling:
+            open_ai_api_call = (
+                "claude" not in self.model_name.lower()
+            )  # False for Anthropic model, True for the rest
+            self._extra_params.update(
+                init_function_calling(
+                    labels=labels,
+                    tool_choices=tool_choices,
+                    open_ai_api_call=open_ai_api_call,
+                )
+            )
+
+    def predict(self, text: str) -> Dict:
+        prompt = self.format_prompt(text)
+        generated_completion = self.language_model_calling(prompt=prompt)
+
+        final_prediction = self.extract_final_prediction(generated_completion)
+
+        return {"prediction": final_prediction}
+
+    @staticmethod
+    def format_prompt(text: str) -> List:
+        return [
+            {
+                "role": "user",
+                "content": [
+                    {"type": "text", "text": text},
+                ],
+            }
+        ]
+
+    def language_model_calling(self, prompt: List):
+        return self._try_again(prompt=prompt)
+
+    def _try_again(self, prompt: List, retries: int = 0):
+        generated_completion = None
+        if retries > self.max_retries:
+            logging_message = f"Max retries exceeded: {retries}."
+            logging.warning(logging_message)
+            self._max_retries_counter += 1
+            return generated_completion
+        try:
+            generated_completion = self._inner_generate_fn(prompt=prompt)
+            return generated_completion
+        except:
+            time.sleep(5)
+            self._try_again(prompt=prompt, retries=retries + 1)
+
+    def extract_final_prediction(self, generated_completion) -> str:
+        if "claude" in self.model_name.lower():
+            if generated_completion is None:
+                final_prediction = None
+            elif generated_completion.content is None:
+                final_prediction = None
+            elif generated_completion.content[0].input is None:
+                final_prediction = None
+            else:
+                try:
+                    final_prediction = generated_completion.content[0].input.get(
+                        "category"
+                    )
+                except:
+                    # Case where the prediction is not a proper dictionary.
+                    final_prediction = generated_completion.content[0].input
+        else:
+            # Something the completion is incomplete, thus we validate that components are there.
+            if generated_completion is None:
+                final_prediction = None
+            elif generated_completion.choices is None:
+                final_prediction = None
+            elif generated_completion.choices[0].message is None:
+                final_prediction = None
+            elif generated_completion.choices[0].message.tool_calls is None:
+                if generated_completion.choices[0].message.content is None:
+                    final_prediction = None
+                else:
+                    # No tools call, but potentially a response in the raw message content.
+                    final_prediction = (
+                        generated_completion.choices[0]
+                        .message.content.strip()
+                        .replace(")", "")
+                        .strip()
+                    )
+            else:
+                prediction = (
+                    generated_completion.choices[0]
+                    .message.tool_calls[0]
+                    .function.arguments
+                )
+                try:
+                    final_prediction = eval(prediction).get("category")
+                except:
+                    # Case where the prediction is not a proper dictionary.
+                    final_prediction = prediction
+
+        if final_prediction is None:
+            # Case were final prediction is None, thus we return -1 to be able to be converted
+            # as int if necessary (infer-case) or left as string (generate-case).
+            # Thus, in both case, it will not yield better results.
+            self._none_prediction_counter += 1
+            final_prediction = f"{NA_VALUE}"
+        elif "La réponse est" in final_prediction or ":" in final_prediction:
+            # To handle case where the LLM return the premise to the last query.
+            final_prediction = final_prediction.split(":")[-1].strip().replace(" ", "")
+        # Cases where the response is accompanied by other string elements, but it should be a single digit.
+        elif "0" in final_prediction:
+            final_prediction = "0"
+        elif "1" in final_prediction:
+            final_prediction = "1"
+        elif "2" in final_prediction:
+            final_prediction = "2"
+        elif "3" in final_prediction:
+            final_prediction = "3"
+        elif "4" in final_prediction:
+            final_prediction = "4"
+        elif "5" in final_prediction:
+            final_prediction = "5"
+        elif "6" in final_prediction:
+            final_prediction = "6"
+        elif "7" in final_prediction:
+            final_prediction = "7"
+        elif "8" in final_prediction:
+            final_prediction = "8"
+        elif "9" in final_prediction:
+            final_prediction = "9"
+        elif "10" in final_prediction:
+            final_prediction = "10"
+        elif "11" in final_prediction:
+            final_prediction = "11"
+
+        return final_prediction
+
+    def print_none(self) -> None:
+        if self._none_prediction_counter > 0:
+            logging_message = f"Number of None: {self._none_prediction_counter}."
+            logging.warning(logging_message)
+        if self._max_retries_counter > 0:
+            logging_message = f"Number of max retries exceeded occurrence: {self._max_retries_counter}."
+            logging.warning(logging_message)

cole/language_model/open_ai_wrapper.py

@@ -0,0 +1,32 @@
+from typing import Dict, List, Union
+
+from openai import OpenAI
+
+from cole.language_model.open_ai_api_lm_wrapper import OpenAIAPILMWrapper
+
+
+class OpenAIWrapper(OpenAIAPILMWrapper):
+    def __init__(
+        self,
+        model_name: str,
+        api_key: str,
+        extra_params: Dict,
+        use_function_calling: bool,
+        base_url: Union[None, str] = None,
+        timeout: int = 480,
+    ):
+        super().__init__(
+            model_name=model_name,
+            extra_params=extra_params,
+            use_function_calling=use_function_calling,
+        )
+        self.client = OpenAI(api_key=api_key, base_url=base_url, timeout=timeout)
+
+    def _inner_generate_fn(self, prompt: List):
+        return self.client.chat.completions.create(
+            model=self.model_name,
+            messages=prompt,
+            n=1,
+            stream=False,
+            **self._extra_params,
+        )

cole/language_model/private_language_model_factory.py

@@ -0,0 +1,117 @@
+import os
+from typing import Union
+
+from predictions.all_llms import private_llm
+from cole.language_model.anthropic_wrapper import AnthropicWrapper
+from cole.language_model.cohere_wrapper import CohereWrapper
+from cole.language_model.deepseek_wrapper import DeepSeekWrapper
+from cole.language_model.mistral_wrapper import MistralWrapper
+from cole.language_model.open_ai_wrapper import OpenAIWrapper
+from cole.language_model.xai_wrapper import XAIWrapper
+
+
+def get_api_key(model_name: str) -> Union[str, None]:
+    if model_name in private_llm["openai"]:
+        key_name = "openai_api_key"
+    elif model_name in private_llm["anthropic"]:
+        key_name = "anthropic_token"
+    elif model_name in private_llm["deepseek"]:
+        key_name = "deepseek_token"
+    elif model_name in private_llm["mistral"]:
+        key_name = "mistral_token"
+    elif model_name in private_llm["xai"]:
+        key_name = "XAI_API_KEY"
+    elif model_name in private_llm["openrouter"]:
+        key_name = "open_route_api_key"
+    elif model_name in private_llm["cohere"]:
+        key_name = "cohere_api_key"
+    else:
+        raise ValueError(f"Model name {model_name} not found.")
+
+    api_key = os.getenv(key_name, None)
+
+    if api_key is None:
+        raise ValueError(f"API key {key_name} not found.")
+    return api_key
+
+
+def private_language_model_factory(model_name):
+    if model_name in private_llm["all"]:
+        api_key = get_api_key(model_name)
+
+        if model_name in private_llm["openai"]:
+            if "o1" in model_name and not "o1-mini" in model_name or "o3" in model_name:
+                extra_params = {
+                    "reasoning_effort": "low"
+                }  # Otherwise take too many tokens and stop the process.
+                if "o3" in model_name:
+                    extra_params.update({"max_completion_tokens": 4096})
+
+            else:
+                extra_params = {}
+            if "mini" in model_name:
+                use_function_calling = False
+            else:
+                use_function_calling = True
+            model = OpenAIWrapper(
+                model_name=model_name,
+                api_key=api_key,
+                extra_params=extra_params,
+                use_function_calling=use_function_calling,
+            )
+        elif model_name in private_llm["anthropic"]:
+            extra_params = {"max_tokens": 5012}
+            model = AnthropicWrapper(
+                model_name=model_name, api_key=api_key, extra_params=extra_params
+            )
+        elif model_name in private_llm["deepseek"]:
+            extra_params = {"timeout": 120}
+            # DeepSeek reasoner does not support function calling
+            use_function_calling = model_name == "deepseek-reasoner"
+            model = DeepSeekWrapper(
+                model_name=model_name,
+                api_key=api_key,
+                extra_params=extra_params,
+                use_function_calling=use_function_calling,
+            )
+        elif model_name in private_llm["xai"]:
+            extra_params = {}
+            model = XAIWrapper(
+                model_name=model_name, api_key=api_key, extra_params=extra_params
+            )
+        elif model_name in private_llm["mistral"]:
+            extra_params = {}
+            model = MistralWrapper(
+                model_name=model_name, api_key=api_key, extra_params=extra_params
+            )
+        elif model_name in private_llm["openrouter"]:
+            extra_params = {}
+            use_function_calling = True
+            model = OpenAIWrapper(
+                model_name=model_name,
+                api_key=api_key,
+                extra_params=extra_params,
+                use_function_calling=use_function_calling,
+                base_url="https://openrouter.ai/api/v1",
+                timeout=480,
+            )
+        elif model_name in private_llm["cohere"]:
+            extra_params = {}
+            if "c4ai-aya" in model_name:
+                use_function_calling = False
+            else:
+                use_function_calling = True
+            model = CohereWrapper(
+                model_name=model_name,
+                api_key=api_key,
+                extra_params=extra_params,
+                use_function_calling=use_function_calling,
+                base_url="https://api.cohere.ai/compatibility/v1",
+                timeout=480,
+            )
+        else:
+            raise NotImplementedError("Not implemented yet.")
+    else:
+        raise ValueError(f"Model name {model_name} not found.")
+
+    return model

cole/language_model/private_lm.py

@@ -0,0 +1,64 @@
+from typing import Union, List
+
+from datasets import Dataset
+from datasets.formatting.formatting import LazyRow
+
+from cole.language_model.language_model_abstraction import LanguageModel
+from cole.language_model.private_language_model_factory import (
+    private_language_model_factory,
+)
+from cole.task.task import Task, TaskType
+
+
+class RemoteLLMModel(LanguageModel):
+    """
+    LLM Model based on private remote LLM provider (e.g. OpenAI) and pipeline mechanism for inference.
+    """
+
+    def generate(self, rows: LazyRow) -> Union[str, List[str]]:
+        try:
+            generate = self.model.predict(rows["text"])
+        except:
+            generate = "NA"
+            print("Generated a NA when model inference.")
+        return generate
+
+    def infer(self, rows: LazyRow) -> Union[str, List[str]]:
+        try:
+            infer = self.model.predict(rows["text"])
+        except:
+            infer = "NA"
+            print("Generated a NA when model inference.")
+        return infer
+
+    def __init__(
+        self,
+        model_name: str,
+        token: Union[str, None] = None,
+    ):
+        super().__init__(model_name)
+        self._model_name = model_name
+        self._token = token
+
+        self.model = private_language_model_factory(model_name=self._model_name)
+
+    def predict(self, evaluation_dataset: Dataset, task: Task) -> List:
+        if task.task_type == TaskType.INFERENCE:
+            labels = task.dataset.possible_ground_truths
+            self.model.init_function_calling(
+                labels, tool_choices=self.model.tool_choices
+            )
+            inference_fn = self.infer
+        else:
+            inference_fn = self.generate
+
+        process_dataset = evaluation_dataset.map(
+            inference_fn,
+            batched=False,
+            desc=f"Running evaluation for task: {task.task_name}",
+            remove_columns="text",
+        )
+
+        self.model.print_none()
+
+        return list(process_dataset["prediction"])

cole/language_model/xai_wrapper.py

@@ -0,0 +1,19 @@
+from typing import Dict, List
+
+from openai import OpenAI
+
+from cole.language_model.open_ai_api_lm_wrapper import OpenAIAPILMWrapper
+
+
+class XAIWrapper(OpenAIAPILMWrapper):
+    def __init__(self, model_name: str, api_key: str, extra_params: Dict):
+        super().__init__(model_name=model_name, extra_params=extra_params)
+        self.client = OpenAI(api_key=api_key, base_url="https://api.x.ai/v1")
+
+    def _inner_generate_fn(self, prompt: List):
+        return self.client.chat.completions.create(
+            model=self.model_name,
+            messages=prompt,
+            stream=False,
+            **self._extra_params,
+        )

cole/metrics/fquad_metric.py

@@ -0,0 +1,108 @@
+import re
+import string
+from collections import Counter
+from typing import Dict, List
+
+from cole.metrics.metrics_wrapper import Metric
+
+
+def normalize_answer(answer: str) -> str:
+    """
+    Lower text and remove punctuation, articles and extra whitespace.
+    Based on the SQUAD official metric: https://huggingface.co/spaces/evaluate-metric/squad
+    """
+
+    def remove_articles(text):
+        # Articles must be removed BEFORE punctuation, because the elided form
+        # "l'" carries its apostrophe; once `remove_punc` strips it, "l'eau" becomes
+        # "leau" and can no longer be matched. Order matters in French.
+        return re.sub(r"\b(le|la|l'|du|des|aux|un|une)\b", " ", text)
+
+    def white_space_fix(text):
+        return " ".join(text.split())
+
+    def remove_punc(text):
+        exclude = set(string.punctuation) | {"«", "»", "’", "“", "”"}
+        apostrophes = {"'", "’", "‘"}
+        # Delete apostrophes (as in the original SQuAD metric), but replace other
+        # punctuation with spaces to avoid gluing tokens together (e.g. in math expressions).
+        parts = []
+        for i, ch in enumerate(text):
+            if ch in exclude:
+                if ch in apostrophes:
+                    continue
+                # Keep minus sign if it's followed by a digit (negative number)
+                if ch == "-" and i + 1 < len(text) and text[i + 1].isdigit():
+                    parts.append(ch)
+                # Keep dot if it's between two digits (decimal point)
+                elif (
+                    ch == "."
+                    and i > 0
+                    and text[i - 1].isdigit()
+                    and i + 1 < len(text)
+                    and text[i + 1].isdigit()
+                ):
+                    parts.append(ch)
+                else:
+                    parts.append(" ")
+            else:
+                parts.append(ch)
+        return "".join(parts)
+
+    answer = str(answer).lower()
+    # Normalize spaces around apostrophes (e.g., "l' eau" or "l 'eau" -> "l'eau")
+    answer = re.sub(r"\s*['’‘]\s*", "'", answer)
+    return white_space_fix(remove_punc(remove_articles(answer)))
+
+
+def f1_score(prediction: str, ground_truth: str) -> float:
+    prediction_tokens = normalize_answer(prediction).split()
+    ground_truth_tokens = normalize_answer(ground_truth).split()
+    common = Counter(prediction_tokens) & Counter(ground_truth_tokens)
+    num_same = sum(common.values())
+    if num_same == 0:
+        return 0.0
+    precision = 1.0 * num_same / len(prediction_tokens)
+    recall = 1.0 * num_same / len(ground_truth_tokens)
+    f1 = (2 * precision * recall) / (precision + recall)
+    return f1
+
+
+def exact_match_score(prediction: str, ground_truth: str) -> float:
+    return normalize_answer(prediction) == normalize_answer(ground_truth)
+
+
+def metric_max_over_ground_truths(metric_fn, prediction, ground_truths):
+    if not ground_truths:
+        # No reference answer available — neither exact match nor F1 can be positive.
+        return 0.0
+    scores_for_ground_truths = []
+    for ground_truth in ground_truths:
+        score = metric_fn(prediction, ground_truth)
+        scores_for_ground_truths.append(score)
+    return max(scores_for_ground_truths)
+
+
+def compute_score(predictions: List, references: List) -> Dict:
+    f1 = exact_match = total = 0
+    for prediction, reference in zip(predictions, references):
+        total += 1
+
+        ground_truths = reference["text"]
+        exact_match += metric_max_over_ground_truths(
+            exact_match_score, prediction, ground_truths
+        )
+        f1 += metric_max_over_ground_truths(f1_score, prediction, ground_truths)
+
+    if total == 0:
+        return {"exact_match": 0.0, "f1": 0.0}
+    exact_match = 100.0 * exact_match / total
+    f1 = 100.0 * f1 / total
+
+    return {"exact_match": exact_match, "f1": f1}
+
+
+class FQuAD(Metric):
+    def compute(self, predictions: List, references: List) -> Dict:
+        score = compute_score(predictions=predictions, references=references)
+        return score