---
license: apache-2.0
tags:
- energy
- hpc
- tabular-regression
- gpt-nl
- sustainability
library_name: gptnl-energy
---

# GPT-NL Energy Prediction Models

Trained energy prediction models for the GPT-NL data curation pipeline.
These models estimate the energy consumption of each pipeline stage
(data splitting → string normalization → heuristic filtering →
toxic language detection → deduplication) running on the Snellius
supercomputer.

## Usage

```bash
pip install git+https://github.com/kruuusher13/gptnl-energy-estimation-ekf.git huggingface_hub

# Download and use a model
from huggingface_hub import hf_hub_download
from gptnl_energy.models import get_model

model_path = hf_hub_download('GPT-NL/gptnl-energy-models', 'linear_energy.joblib')
model = get_model('linear')
model.load_fits(model_path)

# Predict energy for 400k documents
result = model.predict(n=400000, corpus='american_stories')
print(f'{result["total_j"]/1e6:.2f} MJ')
```

## Available Models

| File | Type | Target | Description |
|---|---|---|---|
| `ols_fits_with_dedup.json` | sklearn | energy_j (Joules) | Calibrated per-stage OLS coefficients (production calibration incl. deduplication) — the default fits used by `gptnl-energy forecast` and `monitor` |
| `linear_energy.joblib` | sklearn | energy_j (Joules) | Per-stage OLS linear model (physics baseline): E = c0 + c1*n per pipeline stage |
| `linear_time.joblib` | sklearn | — | — |
| `ridge_energy.joblib` | sklearn | — | — |
| `ridge_time.joblib` | sklearn | — | — |
| `gbm_energy.joblib` | sklearn | energy_j (Joules) | Histogram Gradient Boosting — best ML model for cross-corpus energy transfer |
| `gbm_time.joblib` | sklearn | — | — |
| `mlp_energy.joblib` | sklearn | — | — |
| `mlp_time.joblib` | sklearn | — | — |
| `ftt_energy.pt` | PyTorch | energy_j (Joules) | FT-Transformer — neural tabular model with feature tokenization + stage embedding |
| `ftt_time.pt` | PyTorch | — | — |
| `kalman_transformer_energy.pt` | PyTorch | total_energy_j (whole pipeline, cold start) | FT-Transformer trained on Kalman filter trajectories for cold whole-run prediction |

## Methodology

Three prediction layers:
1. **Physics**: Per-stage linear model E = c0 + c1·n (OLS, calibrated from sample runs)
2. **Learned g**: Coefficient predictor for unseen corpora (GBM, MLP, FT-Transformer)
3. **Kalman filter (EKF)**: Online estimator blending model prediction with live telemetry

See the [GPT-NL Energy repo](https://github.com/GPT-NL/gptnl-energy) for the full pipeline.
## Source and thesis

- Code and data: [kruuusher13/gptnl-energy-estimation-ekf](https://github.com/kruuusher13/gptnl-energy-estimation-ekf)
- Thesis: *GPT-NL: Complexity-Aware Energy Estimation of the Data Curation Pipeline via Extended Kalman Filtering*, R. Malik, MSc Applied Data Science, Utrecht University, 2026 (in the repository under `paper/thesis.pdf`)
- Pipeline this predicts: [GPT-NL data-curation-pipeline](https://github.com/GPT-NL/data-curation-pipeline)

All coefficients and evaluation numbers are regenerable from the measurement data with the scripts in `paper/code/`.