| --- |
| license: apache-2.0 |
| tags: |
| - energy |
| - hpc |
| - tabular-regression |
| - gpt-nl |
| - sustainability |
| library_name: gptnl-energy |
| --- |
| |
| # GPT-NL Energy Prediction Models |
|
|
| Trained energy prediction models for the GPT-NL data curation pipeline. |
| These models estimate the energy consumption of each pipeline stage |
| (data splitting → string normalization → heuristic filtering → |
| toxic language detection → deduplication) running on the Snellius |
| supercomputer. |
|
|
| ## Usage |
|
|
| ```bash |
| pip install git+https://github.com/kruuusher13/gptnl-energy-estimation-ekf.git huggingface_hub |
| |
| # Download and use a model |
| from huggingface_hub import hf_hub_download |
| from gptnl_energy.models import get_model |
| |
| model_path = hf_hub_download('GPT-NL/gptnl-energy-models', 'linear_energy.joblib') |
| model = get_model('linear') |
| model.load_fits(model_path) |
| |
| # Predict energy for 400k documents |
| result = model.predict(n=400000, corpus='american_stories') |
| print(f'{result["total_j"]/1e6:.2f} MJ') |
| ``` |
|
|
| ## Available Models |
|
|
| | File | Type | Target | Description | |
| |---|---|---|---| |
| | `ols_fits_with_dedup.json` | sklearn | energy_j (Joules) | Calibrated per-stage OLS coefficients (production calibration incl. deduplication) — the default fits used by `gptnl-energy forecast` and `monitor` | |
| | `linear_energy.joblib` | sklearn | energy_j (Joules) | Per-stage OLS linear model (physics baseline): E = c0 + c1*n per pipeline stage | |
| | `linear_time.joblib` | sklearn | — | — | |
| | `ridge_energy.joblib` | sklearn | — | — | |
| | `ridge_time.joblib` | sklearn | — | — | |
| | `gbm_energy.joblib` | sklearn | energy_j (Joules) | Histogram Gradient Boosting — best ML model for cross-corpus energy transfer | |
| | `gbm_time.joblib` | sklearn | — | — | |
| | `mlp_energy.joblib` | sklearn | — | — | |
| | `mlp_time.joblib` | sklearn | — | — | |
| | `ftt_energy.pt` | PyTorch | energy_j (Joules) | FT-Transformer — neural tabular model with feature tokenization + stage embedding | |
| | `ftt_time.pt` | PyTorch | — | — | |
| | `kalman_transformer_energy.pt` | PyTorch | total_energy_j (whole pipeline, cold start) | FT-Transformer trained on Kalman filter trajectories for cold whole-run prediction | |
|
|
| ## Methodology |
|
|
| Three prediction layers: |
| 1. **Physics**: Per-stage linear model E = c0 + c1·n (OLS, calibrated from sample runs) |
| 2. **Learned g**: Coefficient predictor for unseen corpora (GBM, MLP, FT-Transformer) |
| 3. **Kalman filter (EKF)**: Online estimator blending model prediction with live telemetry |
|
|
| See the [GPT-NL Energy repo](https://github.com/GPT-NL/gptnl-energy) for the full pipeline. |
| ## Source and thesis |
|
|
| - Code and data: [kruuusher13/gptnl-energy-estimation-ekf](https://github.com/kruuusher13/gptnl-energy-estimation-ekf) |
| - Thesis: *GPT-NL: Complexity-Aware Energy Estimation of the Data Curation Pipeline via Extended Kalman Filtering*, R. Malik, MSc Applied Data Science, Utrecht University, 2026 (in the repository under `paper/thesis.pdf`) |
| - Pipeline this predicts: [GPT-NL data-curation-pipeline](https://github.com/GPT-NL/data-curation-pipeline) |
|
|
| All coefficients and evaluation numbers are regenerable from the measurement data with the scripts in `paper/code/`. |
|
|
