File size: 3,144 Bytes
0df38d9 8e07ebb 5b25191 8e07ebb 5b25191 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: apache-2.0
tags:
- energy
- hpc
- tabular-regression
- gpt-nl
- sustainability
library_name: gptnl-energy
---
# GPT-NL Energy Prediction Models
Trained energy prediction models for the GPT-NL data curation pipeline.
These models estimate the energy consumption of each pipeline stage
(data splitting → string normalization → heuristic filtering →
toxic language detection → deduplication) running on the Snellius
supercomputer.
## Usage
```bash
pip install git+https://github.com/kruuusher13/gptnl-energy-estimation-ekf.git huggingface_hub
# Download and use a model
from huggingface_hub import hf_hub_download
from gptnl_energy.models import get_model
model_path = hf_hub_download('GPT-NL/gptnl-energy-models', 'linear_energy.joblib')
model = get_model('linear')
model.load_fits(model_path)
# Predict energy for 400k documents
result = model.predict(n=400000, corpus='american_stories')
print(f'{result["total_j"]/1e6:.2f} MJ')
```
## Available Models
| File | Type | Target | Description |
|---|---|---|---|
| `ols_fits_with_dedup.json` | sklearn | energy_j (Joules) | Calibrated per-stage OLS coefficients (production calibration incl. deduplication) — the default fits used by `gptnl-energy forecast` and `monitor` |
| `linear_energy.joblib` | sklearn | energy_j (Joules) | Per-stage OLS linear model (physics baseline): E = c0 + c1*n per pipeline stage |
| `linear_time.joblib` | sklearn | — | — |
| `ridge_energy.joblib` | sklearn | — | — |
| `ridge_time.joblib` | sklearn | — | — |
| `gbm_energy.joblib` | sklearn | energy_j (Joules) | Histogram Gradient Boosting — best ML model for cross-corpus energy transfer |
| `gbm_time.joblib` | sklearn | — | — |
| `mlp_energy.joblib` | sklearn | — | — |
| `mlp_time.joblib` | sklearn | — | — |
| `ftt_energy.pt` | PyTorch | energy_j (Joules) | FT-Transformer — neural tabular model with feature tokenization + stage embedding |
| `ftt_time.pt` | PyTorch | — | — |
| `kalman_transformer_energy.pt` | PyTorch | total_energy_j (whole pipeline, cold start) | FT-Transformer trained on Kalman filter trajectories for cold whole-run prediction |
## Methodology
Three prediction layers:
1. **Physics**: Per-stage linear model E = c0 + c1·n (OLS, calibrated from sample runs)
2. **Learned g**: Coefficient predictor for unseen corpora (GBM, MLP, FT-Transformer)
3. **Kalman filter (EKF)**: Online estimator blending model prediction with live telemetry
See the [GPT-NL Energy repo](https://github.com/GPT-NL/gptnl-energy) for the full pipeline.
## Source and thesis
- Code and data: [kruuusher13/gptnl-energy-estimation-ekf](https://github.com/kruuusher13/gptnl-energy-estimation-ekf)
- Thesis: *GPT-NL: Complexity-Aware Energy Estimation of the Data Curation Pipeline via Extended Kalman Filtering*, R. Malik, MSc Applied Data Science, Utrecht University, 2026 (in the repository under `paper/thesis.pdf`)
- Pipeline this predicts: [GPT-NL data-curation-pipeline](https://github.com/GPT-NL/data-curation-pipeline)
All coefficients and evaluation numbers are regenerable from the measurement data with the scripts in `paper/code/`.
|