exp03c_ppo_3x4_backplay_relaxed: checkpoint, config, evaluation files, model card
Browse files
README.md
CHANGED
|
@@ -1,149 +1,149 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mit
|
| 3 |
-
library_name: stable-baselines3
|
| 4 |
-
pipeline_tag: reinforcement-learning
|
| 5 |
-
tags:
|
| 6 |
-
- 4d-snake
|
| 7 |
-
- 4d-snake-3x4
|
| 8 |
-
- snake
|
| 9 |
-
- reinforcement-learning
|
| 10 |
-
- deep-reinforcement-learning
|
| 11 |
-
- stable-baselines3
|
| 12 |
-
- sb3-contrib
|
| 13 |
-
- maskable-ppo
|
| 14 |
-
- backplay-curriculum
|
| 15 |
-
model-index:
|
| 16 |
-
- name: 4d-snake-exp03c-ppo-3x4-backplay-relaxed
|
| 17 |
-
results:
|
| 18 |
-
- task:
|
| 19 |
-
type: reinforcement-learning
|
| 20 |
-
name: reinforcement-learning
|
| 21 |
-
dataset:
|
| 22 |
-
name: 4d-snake-3x4
|
| 23 |
-
type: 4d-snake-3x4
|
| 24 |
-
metrics:
|
| 25 |
-
- type: completion_rate
|
| 26 |
-
name: completion rate (deterministic, 100 episodes x 3 seeds)
|
| 27 |
-
value: 0.000 +/- 0.000
|
| 28 |
-
verified: false
|
| 29 |
-
- type: mean_reward
|
| 30 |
-
name: mean episode return (deterministic)
|
| 31 |
-
value: 41.75
|
| 32 |
-
verified: false
|
| 33 |
-
- type: fill
|
| 34 |
-
name: mean final fill (deterministic)
|
| 35 |
-
value: 0.542
|
| 36 |
-
verified: false
|
| 37 |
-
---
|
| 38 |
-
|
| 39 |
-
# 4d-snake-exp03c-ppo-3x4-backplay-relaxed
|
| 40 |
-
|
| 41 |
-
A `512x512` MLP policy for **4-dimensional snake on the 3^4 board** (81 cells, 8 moves): MaskablePPO trained from scratch, Backplay reverse curriculum (gate 0.8, window 8), 30,000,000 environment steps. From a length-1 start it completes the board in 0.0 % of deterministic episodes, evaluated with the protocol of [docs/evaluation.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/docs/evaluation.md) (100 episodes x 3 seeds, masked `evaluate_policy`).
|
| 42 |
-
|
| 43 |
-
**Negative result.** This network never fills the board from the true start (mean final fill 0.542). It is published so the failure is reproducible; the write-up linked below analyses why.
|
| 44 |
-
|
| 45 |
-
## Results (`eval/summary.json`)
|
| 46 |
-
|
| 47 |
-
| mode | completion +- std | mean fill | steps to complete | won within 4C |
|
| 48 |
-
|---|---|---|---|---|
|
| 49 |
-
| deterministic (argmax) | 0.000 +- 0.000 | 0.542 | never | 0.000 |
|
| 50 |
-
| sampling | 0.000 +- 0.000 | 0.545 | never | 0.000 |
|
| 51 |
-
|
| 52 |
-
## How to use
|
| 53 |
-
|
| 54 |
-
The observation is this repository's `4*C + 2` float vector and the action space its `2*ndim` masked moves ([docs/game_rules.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/docs/game_rules.md)), so the checkpoint runs inside `snake4d`'s environment:
|
| 55 |
-
|
| 56 |
-
```bash
|
| 57 |
-
git clone https://github.com/BurnyCoder/4d-snake-rl.git && cd 4d-snake-rl && uv sync
|
| 58 |
-
hf download BurnyCoder/4d-snake-exp03c-ppo-3x4-backplay-relaxed best_model.zip --local-dir weights
|
| 59 |
-
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=3 --set ndim=4
|
| 60 |
-
```
|
| 61 |
-
|
| 62 |
-
```python
|
| 63 |
-
# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
|
| 64 |
-
from sb3_contrib import MaskablePPO
|
| 65 |
-
from sb3_contrib.common.maskable.utils import get_action_masks
|
| 66 |
-
from snake4d.config import Config
|
| 67 |
-
from snake4d.vec_env import make_env
|
| 68 |
-
|
| 69 |
-
cfg = Config(size=3, ndim=4)
|
| 70 |
-
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
|
| 71 |
-
env = make_env(cfg, 1, 0) # one board; observation shape (1, 4*C + 2)
|
| 72 |
-
obs = env.reset()
|
| 73 |
-
masks = get_action_masks(env) # the legal moves, one row per board
|
| 74 |
-
action, _ = model.predict(obs, action_masks=masks, deterministic=True)
|
| 75 |
-
```
|
| 76 |
-
|
| 77 |
-
## Training
|
| 78 |
-
|
| 79 |
-
- Phase `train`; experiment file `experiments/exp03c_ppo_3x4_backplay_relaxed.env`; write-up: https://github.com/BurnyCoder/4d-snake-rl/blob/main/reports/experiments/exp03_ppo_3x4.md.
|
| 80 |
-
- Resolved configuration (`config.json`):
|
| 81 |
-
|
| 82 |
-
```json
|
| 83 |
-
{
|
| 84 |
-
"size": 3,
|
| 85 |
-
"ndim": 4,
|
| 86 |
-
"idle_mult": 4,
|
| 87 |
-
"r_food": 1.0,
|
| 88 |
-
"r_death": -1.0,
|
| 89 |
-
"r_win": 10.0,
|
| 90 |
-
"r_step": -0.001,
|
| 91 |
-
"shaping_coef": 0.0,
|
| 92 |
-
"n_envs": 2048,
|
| 93 |
-
"total_timesteps": 30000000,
|
| 94 |
-
"n_steps": 64,
|
| 95 |
-
"batch_size": 8192,
|
| 96 |
-
"n_epochs": 4,
|
| 97 |
-
"gamma": 0.99,
|
| 98 |
-
"gae_lambda": 0.95,
|
| 99 |
-
"lr_start": 0.0003,
|
| 100 |
-
"lr_end": 1e-05,
|
| 101 |
-
"clip_start": 0.2,
|
| 102 |
-
"clip_end": 0.05,
|
| 103 |
-
"ent_coef": 0.01,
|
| 104 |
-
"vf_coef": 0.5,
|
| 105 |
-
"max_grad_norm": 0.5,
|
| 106 |
-
"target_kl": 0.03,
|
| 107 |
-
"net_width": 512,
|
| 108 |
-
"device": "auto",
|
| 109 |
-
"torch_threads": 8,
|
| 110 |
-
"seed": 0,
|
| 111 |
-
"curriculum": 1,
|
| 112 |
-
"curriculum_window": 8,
|
| 113 |
-
"curriculum_delta": 4,
|
| 114 |
-
"curriculum_rho": 0.8,
|
| 115 |
-
"curriculum_min_eps": 500,
|
| 116 |
-
"p_true_start": 0.2,
|
| 117 |
-
"eval_episodes": 100,
|
| 118 |
-
"eval_every": 1310720,
|
| 119 |
-
"ckpt_every": 5242880,
|
| 120 |
-
"eval_seeds": "0,1,2",
|
| 121 |
-
"bench_steps": 200000,
|
| 122 |
-
"runs_dir": "runs",
|
| 123 |
-
"run_name": "exp03c_ppo_3x4_backplay_relaxed",
|
| 124 |
-
"model_path": "",
|
| 125 |
-
"policy": "route"
|
| 126 |
-
}
|
| 127 |
-
```
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
## Provenance
|
| 134 |
-
|
| 135 |
-
- Code: https://github.com/BurnyCoder/4d-snake-rl at commit `9dfaa8a6f662d57fc8a02ee67a15efa3cbefab6c`.
|
| 136 |
-
- Library versions (`versions.json`): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
|
| 137 |
-
- `eval/summary.json` and `eval/eval_episodes.csv` are the files the repository's reports quote; every evaluated network is compared in [reports/networks.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/reports/networks.md).
|
| 138 |
-
- Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741
|
| 139 |
-
|
| 140 |
-
## Files
|
| 141 |
-
|
| 142 |
-
- `best_model.zip`: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
|
| 143 |
-
- `config.json`, `versions.json`: the run's resolved configuration and environment.
|
| 144 |
-
- `eval/`: evaluation summary and one row per evaluation episode.
|
| 145 |
-
- `train/progress.csv`: the SB3 training log; `figures/`: the learning curves.
|
| 146 |
-
|
| 147 |
-
## Licence
|
| 148 |
-
|
| 149 |
-
MIT, like the repository.
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: stable-baselines3
|
| 4 |
+
pipeline_tag: reinforcement-learning
|
| 5 |
+
tags:
|
| 6 |
+
- 4d-snake
|
| 7 |
+
- 4d-snake-3x4
|
| 8 |
+
- snake
|
| 9 |
+
- reinforcement-learning
|
| 10 |
+
- deep-reinforcement-learning
|
| 11 |
+
- stable-baselines3
|
| 12 |
+
- sb3-contrib
|
| 13 |
+
- maskable-ppo
|
| 14 |
+
- backplay-curriculum
|
| 15 |
+
model-index:
|
| 16 |
+
- name: 4d-snake-exp03c-ppo-3x4-backplay-relaxed
|
| 17 |
+
results:
|
| 18 |
+
- task:
|
| 19 |
+
type: reinforcement-learning
|
| 20 |
+
name: reinforcement-learning
|
| 21 |
+
dataset:
|
| 22 |
+
name: 4d-snake-3x4
|
| 23 |
+
type: 4d-snake-3x4
|
| 24 |
+
metrics:
|
| 25 |
+
- type: completion_rate
|
| 26 |
+
name: completion rate (deterministic, 100 episodes x 3 seeds)
|
| 27 |
+
value: 0.000 +/- 0.000
|
| 28 |
+
verified: false
|
| 29 |
+
- type: mean_reward
|
| 30 |
+
name: mean episode return (deterministic)
|
| 31 |
+
value: 41.75
|
| 32 |
+
verified: false
|
| 33 |
+
- type: fill
|
| 34 |
+
name: mean final fill (deterministic)
|
| 35 |
+
value: 0.542
|
| 36 |
+
verified: false
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
# 4d-snake-exp03c-ppo-3x4-backplay-relaxed
|
| 40 |
+
|
| 41 |
+
A `512x512` MLP policy for **4-dimensional snake on the 3^4 board** (81 cells, 8 moves): MaskablePPO trained from scratch, Backplay reverse curriculum (gate 0.8, window 8), 30,000,000 environment steps. From a length-1 start it completes the board in 0.0 % of deterministic episodes, evaluated with the protocol of [docs/evaluation.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/docs/evaluation.md) (100 episodes x 3 seeds, masked `evaluate_policy`).
|
| 42 |
+
|
| 43 |
+
**Negative result.** This network never fills the board from the true start (mean final fill 0.542). It is published so the failure is reproducible; the write-up linked below analyses why.
|
| 44 |
+
|
| 45 |
+
## Results (`eval/summary.json`)
|
| 46 |
+
|
| 47 |
+
| mode | completion +- std | mean fill | steps to complete | won within 4C |
|
| 48 |
+
|---|---|---|---|---|
|
| 49 |
+
| deterministic (argmax) | 0.000 +- 0.000 | 0.542 | never | 0.000 |
|
| 50 |
+
| sampling | 0.000 +- 0.000 | 0.545 | never | 0.000 |
|
| 51 |
+
|
| 52 |
+
## How to use
|
| 53 |
+
|
| 54 |
+
The observation is this repository's `4*C + 2` float vector and the action space its `2*ndim` masked moves ([docs/game_rules.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/docs/game_rules.md)), so the checkpoint runs inside `snake4d`'s environment:
|
| 55 |
+
|
| 56 |
+
```bash
|
| 57 |
+
git clone https://github.com/BurnyCoder/4d-snake-rl.git && cd 4d-snake-rl && uv sync
|
| 58 |
+
hf download BurnyCoder/4d-snake-exp03c-ppo-3x4-backplay-relaxed best_model.zip --local-dir weights
|
| 59 |
+
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=3 --set ndim=4
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
```python
|
| 63 |
+
# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
|
| 64 |
+
from sb3_contrib import MaskablePPO
|
| 65 |
+
from sb3_contrib.common.maskable.utils import get_action_masks
|
| 66 |
+
from snake4d.config import Config
|
| 67 |
+
from snake4d.vec_env import make_env
|
| 68 |
+
|
| 69 |
+
cfg = Config(size=3, ndim=4)
|
| 70 |
+
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
|
| 71 |
+
env = make_env(cfg, 1, 0) # one board; observation shape (1, 4*C + 2)
|
| 72 |
+
obs = env.reset()
|
| 73 |
+
masks = get_action_masks(env) # the legal moves, one row per board
|
| 74 |
+
action, _ = model.predict(obs, action_masks=masks, deterministic=True)
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
+
## Training
|
| 78 |
+
|
| 79 |
+
- Phase `train`; experiment file `experiments/exp03c_ppo_3x4_backplay_relaxed.env`; write-up: https://github.com/BurnyCoder/4d-snake-rl/blob/main/reports/experiments/exp03_ppo_3x4.md.
|
| 80 |
+
- Resolved configuration (`config.json`):
|
| 81 |
+
|
| 82 |
+
```json
|
| 83 |
+
{
|
| 84 |
+
"size": 3,
|
| 85 |
+
"ndim": 4,
|
| 86 |
+
"idle_mult": 4,
|
| 87 |
+
"r_food": 1.0,
|
| 88 |
+
"r_death": -1.0,
|
| 89 |
+
"r_win": 10.0,
|
| 90 |
+
"r_step": -0.001,
|
| 91 |
+
"shaping_coef": 0.0,
|
| 92 |
+
"n_envs": 2048,
|
| 93 |
+
"total_timesteps": 30000000,
|
| 94 |
+
"n_steps": 64,
|
| 95 |
+
"batch_size": 8192,
|
| 96 |
+
"n_epochs": 4,
|
| 97 |
+
"gamma": 0.99,
|
| 98 |
+
"gae_lambda": 0.95,
|
| 99 |
+
"lr_start": 0.0003,
|
| 100 |
+
"lr_end": 1e-05,
|
| 101 |
+
"clip_start": 0.2,
|
| 102 |
+
"clip_end": 0.05,
|
| 103 |
+
"ent_coef": 0.01,
|
| 104 |
+
"vf_coef": 0.5,
|
| 105 |
+
"max_grad_norm": 0.5,
|
| 106 |
+
"target_kl": 0.03,
|
| 107 |
+
"net_width": 512,
|
| 108 |
+
"device": "auto",
|
| 109 |
+
"torch_threads": 8,
|
| 110 |
+
"seed": 0,
|
| 111 |
+
"curriculum": 1,
|
| 112 |
+
"curriculum_window": 8,
|
| 113 |
+
"curriculum_delta": 4,
|
| 114 |
+
"curriculum_rho": 0.8,
|
| 115 |
+
"curriculum_min_eps": 500,
|
| 116 |
+
"p_true_start": 0.2,
|
| 117 |
+
"eval_episodes": 100,
|
| 118 |
+
"eval_every": 1310720,
|
| 119 |
+
"ckpt_every": 5242880,
|
| 120 |
+
"eval_seeds": "0,1,2",
|
| 121 |
+
"bench_steps": 200000,
|
| 122 |
+
"runs_dir": "runs",
|
| 123 |
+
"run_name": "exp03c_ppo_3x4_backplay_relaxed",
|
| 124 |
+
"model_path": "",
|
| 125 |
+
"policy": "route"
|
| 126 |
+
}
|
| 127 |
+
```
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
## Provenance
|
| 134 |
+
|
| 135 |
+
- Code: https://github.com/BurnyCoder/4d-snake-rl at commit `9dfaa8a6f662d57fc8a02ee67a15efa3cbefab6c`.
|
| 136 |
+
- Library versions (`versions.json`): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
|
| 137 |
+
- `eval/summary.json` and `eval/eval_episodes.csv` are the files the repository's reports quote; every evaluated network is compared in [reports/networks.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/reports/networks.md).
|
| 138 |
+
- Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741
|
| 139 |
+
|
| 140 |
+
## Files
|
| 141 |
+
|
| 142 |
+
- `best_model.zip`: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
|
| 143 |
+
- `config.json`, `versions.json`: the run's resolved configuration and environment.
|
| 144 |
+
- `eval/`: evaluation summary and one row per evaluation episode.
|
| 145 |
+
- `train/progress.csv`: the SB3 training log; `figures/`: the learning curves.
|
| 146 |
+
|
| 147 |
+
## Licence
|
| 148 |
+
|
| 149 |
+
MIT, like the repository.
|