| --- |
| library_name: pytorch |
| tags: |
| - decision-transformer |
| - energy-trading |
| - battery-storage |
| - aemo |
| - nem |
| - offline-rl |
| datasets: |
| - mrvictoru/AEMO_simulated_trade |
| new_version: mrvictoru/energydecision-dt-v2-sdp |
| pipeline_tag: reinforcement-learning |
| --- |
| # EnergyDecision-DT-V2 |
|
|
| > **⚠️ OUTDATED — SUPERSeded by the Stage C standalone DT.** |
| > |
| > This model (modern v2 pretrained, 8×768 GQA) was the SOTA as of Jul 2026, |
| > beating PPO on dispatch-matched ($10,138/ep) and standard ($4,630/ep) surfaces. |
| > It has since been superseded by the **Stage C standalone DT** distilled from |
| > an honest SDP-planning teacher, which beats PPO on **all 4 identity surfaces** |
| > and passes the market-impact gate: |
| > |
| > | Surface | Stage C DT (`rtg_mode="auto"`) | This model (v2 pretrained) | PPO | |
| > |---|---|---|---| |
| > | Standard Oct | **$11,573** | $4,991 | $2,353 | |
| > | Dispatch-matched | **$35,320** | $10,138 | $22,530 | |
| > | Expanded broad-2024 | **$34,761** | $4,596 | $19,504 | |
| > | 2025 OOD | **$25,862** | −$694 | $14,320 | |
| > | Impact gate | **2.6–3.0× PPO** | 62–83% identity | — | |
| > |
| > The shipped model is `models/aemo/dt/aemo_dt_sdp_jtsoc_fullcorpus.pt` |
| > (see [energydecision repo](https://github.com/mrvictoru/energydecision)). |
| > This checkpoint is retained for reproducing the Jul 2026 benchmark study. |
| |
| --- |
| |
| ## Model Description |
| |
| **EnergyDecision-DT-V2** is a **Decision Transformer** model trained on simulated battery dispatch data from the Australian Energy Market Operator (AEMO) Frequency Control Ancillary Services (FCAS) market. It models optimal battery dispatch as a sequence prediction problem, conditioning on returns-to-go, observed states, and past actions to predict the next action. |
| |
| The model learns to **dispatch battery energy storage** (charge/discharge) and **bid into 8 FCAS contingency markets** simultaneously, using a modern transformer architecture with Grouped-Query Attention, QK-Norm, SwiGLU activations, and weight-tied embeddings. |
| |
| ### Key Features |
| |
| - **Modern architecture**: Grouped-Query Attention (6 KV heads, 12 Q heads), QK-Norm for training stability, SwiGLU FFN, RMSNorm pre-norm |
| - **Weight tying**: Embedding and prediction layers share weights for parameter efficiency |
| - **Action Space (9-dim)**: |
| - Dim 0: Energy dispatch in [-1, 1] (negative = charge, positive = discharge) |
| - Dims 1-8: FCAS contingency bids in [0, 1] |
| - **State Space (18-dim)**: Normalized market observations including prices, demand, renewables penetration, and battery state-of-charge |
| - **Context Length**: 210 timesteps (looks back ~17.5 hours of 5-minute dispatch intervals) |
| |
| ## Intended Use |
| |
| This model is intended for: |
| |
| - **Research** into offline RL for energy markets |
| - **Simulation** of battery trading strategies in the AEMO FCAS market |
| - **Baseline** for comparing decision transformer approaches against traditional RL |
| |
| It is **not intended for live trading** without further validation, risk management, and regulatory compliance. |
| |
| ## Training Data |
| |
| - **Source**: [AEMO simulated trade dataset](https://huggingface.co/datasets/mrvictoru/AEMO_simulated_trade) |
| - **Size**: 86,412,124 rows after filtering |
| - **Episodes**: 2,401 episodes (after filtering for minimum context length) |
| - **Source policies**: A2C (76.9M rows) + GRPO-DT (11.9M rows) |
| |
| ## Model Architecture |
| |
| ``` |
| DecisionTransformer( |
| (embed_return): Linear(1 -> 768) |
| (embed_state): Linear(18 -> 768) |
| (embed_action): Linear(9 -> 768) |
| (embed_timestep): Embedding(100000 -> 768) |
| (embed_ln): RMSNorm(768) |
| (blocks): 8x ModernBlock( |
| (norm1): RMSNorm(768) |
| (attn): CausalSelfAttention( |
| q_proj: Linear(768 -> 768) # 12 Q heads x 64 head_dim |
| k_proj: Linear(768 -> 384) # 6 KV heads x 64 head_dim |
| v_proj: Linear(768 -> 384) # 6 KV heads x 64 head_dim |
| out_proj: Linear(768 -> 768) |
| qk_norm: RMSNorm(64) per head |
| n_rep: 2 (each KV head serves 2 Q heads -- GQA) |
| ) |
| (norm2): RMSNorm(768) |
| (ffn): SwiGLU(768 -> 3072 -> 768, dropout=0.15) |
| ) |
| (ln_f): RMSNorm(768) |
| (pred_act): Linear(768 -> 9) -> Tanh [tied with embed_act weights] |
| (pred_state): Linear(768 -> 18) [tied with embed_state weights] |
| (pred_return): Linear(768 -> 1) [tied with embed_return weights] |
| ) |
| ``` |
| |
| ### Hyperparameters |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Blocks | 8 | |
| | Hidden dim | 768 | |
| | Attention heads (Q) | 12 | |
| | KV heads (GQA) | 6 | |
| | Context length | 210 | |
| | Dropout | 0.15 | |
| | QK-Norm | Enabled | |
| | Weight tying | Enabled | |
| | State dim | 18 | |
| | Action dim | 9 | |
| | Discount factor | 0.95 | |
| | Return scale | 2.0 | |
|
|
| ## Training Procedure |
|
|
| - **Hardware**: NVIDIA GPU (AMP mixed precision) |
| - **Optimizer**: AdamW (lr=3e-5, weight_decay=1e-4) |
| - **Batch size**: 128 |
| - **Epochs**: 3 |
| - **Gradient clipping**: 1.0 |
| |
| ### Training Metrics |
| |
| | Epoch | Train Loss | Val Loss | Action Loss | |
| |-------|-----------|----------|-------------| |
| | 1 | 0.057152 | 0.019429 | 0.056399 | |
| | 2 | 0.015032 | 0.009449 | 0.014647 | |
| | 3 | **0.008868** | **0.007034** | **0.008644** | |
| |
| ## Benchmark Results (historical — Jul 2026) |
| |
| Evaluated on dispatch-matched (Dalrymple North 8 MWh / 30 MW, Q4 2024 SA1) and standard (5 regions, medium batteries) surfaces: |
| |
| | Model | Standard | DM (rtg=0.5) | DM (rtg=0.0) | |
| |---|---|---|---| |
| | **Modern v2 pretrained (this model)** | **$4,630** | $6,793 | **$10,138** | |
| | PPO reference | $2,353 | $7,757 | — | |
| | Dispatch Dalrymple North | $4,660 | $3,663 | — | |
| |
| ## RTG Calibration (historical) |
| |
| | RTG | Profit/ep | FCAS/ep | |
| |-----|----------|---------| |
| | **0.0** | **$10,138** | $10,068 | |
| | 0.5 | $6,793 | $6,703 | |
| | 1.0 | $6,877 | $6,101 | |
| | 2.0 | $6,329 | $6,092 | |
| |
| ## Usage |
| |
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from decision_transformer import DecisionTransformer |
| |
| model_kwargs = { |
| "state_dim": 18, "act_dim": 9, "n_block": 8, |
| "h_dim": 768, "n_heads": 12, "n_kv_heads": 6, |
| "context_len": 210, "drop_p": 0.15, |
| "max_timestep": 100000, "qk_norm": True, |
| "rope_enabled": False, "tie_weights": True, |
| } |
| |
| model_path = hf_hub_download("mrvictoru/energydecision-dt-v2", "aemo_dt_fcas_model.pt") |
| model = DecisionTransformer(**model_kwargs) |
| model.load_from_checkpoint(model_path) |
| model.eval() |
| ``` |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{energydecision-dt-v2, |
| author = {Victor U}, |
| title = {EnergyDecision-DT-V2: Decision Transformer for AEMO FCAS Battery Trading}, |
| year = {2026}, |
| publisher = {HuggingFace}, |
| howpublished = {\url{https://huggingface.co/mrvictoru/energydecision-dt-v2}}, |
| } |
| ``` |
