File size: 5,316 Bytes
929734b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 | ---
license: mit
library_name: stable-baselines3
pipeline_tag: reinforcement-learning
tags:
- 4d-snake
- 4d-snake-2x4
- snake
- reinforcement-learning
- deep-reinforcement-learning
- stable-baselines3
- sb3-contrib
- maskable-ppo
model-index:
- name: 4d-snake-exp02-ppo-2x4
results:
- task:
type: reinforcement-learning
name: reinforcement-learning
dataset:
name: 4d-snake-2x4
type: 4d-snake-2x4
metrics:
- type: completion_rate
name: completion rate (deterministic, 100 episodes x 3 seeds)
value: 0.877 +/- 0.041
verified: false
- type: mean_reward
name: mean episode return (deterministic)
value: 23.41
verified: false
- type: fill
name: mean final fill (deterministic)
value: 0.987
verified: false
- type: steps_to_complete
name: mean steps to complete (deterministic, won episodes)
value: 34.4
verified: false
---
# 4d-snake-exp02-ppo-2x4
A `512x512` MLP policy for **4-dimensional snake on the 2^4 board** (16 cells, 8 moves): MaskablePPO trained from scratch, no curriculum, 5,000,000 environment steps. From a length-1 start it completes the board in 87.7 % of deterministic episodes, evaluated with the protocol of [docs/evaluation.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/docs/evaluation.md) (100 episodes x 3 seeds, masked `evaluate_policy`).
## Results (`eval/summary.json`)
| mode | completion +- std | mean fill | steps to complete | won within 4C |
|---|---|---|---|---|
| deterministic (argmax) | 0.877 +- 0.041 | 0.987 | 34.4 | 0.877 |
| sampling | 0.853 +- 0.034 | 0.985 | 35.5 | 0.853 |
## How to use
The observation is this repository's `4*C + 2` float vector and the action space its `2*ndim` masked moves ([docs/game_rules.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/docs/game_rules.md)), so the checkpoint runs inside `snake4d`'s environment:
```bash
git clone https://github.com/BurnyCoder/4d-snake-rl.git && cd 4d-snake-rl && uv sync
hf download BurnyCoder/4d-snake-exp02-ppo-2x4 best_model.zip --local-dir weights
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=2 --set ndim=4
```
```python
# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
from sb3_contrib import MaskablePPO
from sb3_contrib.common.maskable.utils import get_action_masks
from snake4d.config import Config
from snake4d.vec_env import make_env
cfg = Config(size=2, ndim=4)
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
env = make_env(cfg, 1, 0) # one board; observation shape (1, 4*C + 2)
obs = env.reset()
masks = get_action_masks(env) # the legal moves, one row per board
action, _ = model.predict(obs, action_masks=masks, deterministic=True)
```
## Training
- Phase `train`; experiment file `experiments/exp02_ppo_2x4.env`; write-up: https://github.com/BurnyCoder/4d-snake-rl/blob/main/reports/experiments/exp02_ppo_2x4.md.
- Resolved configuration (`config.json`):
```json
{
"size": 2,
"ndim": 4,
"idle_mult": 4,
"r_food": 1.0,
"r_death": -1.0,
"r_win": 10.0,
"r_step": -0.001,
"shaping_coef": 0.0,
"n_envs": 1024,
"total_timesteps": 5000000,
"n_steps": 32,
"batch_size": 4096,
"n_epochs": 4,
"gamma": 0.99,
"gae_lambda": 0.95,
"lr_start": 0.0003,
"lr_end": 1e-05,
"clip_start": 0.2,
"clip_end": 0.05,
"ent_coef": 0.01,
"vf_coef": 0.5,
"max_grad_norm": 0.5,
"target_kl": 0.03,
"net_width": 512,
"device": "auto",
"torch_threads": 8,
"seed": 0,
"curriculum": 0,
"curriculum_window": 8,
"curriculum_delta": 0,
"curriculum_rho": 0.2,
"curriculum_min_eps": 200,
"p_true_start": 0.2,
"eval_episodes": 100,
"eval_every": 327680,
"ckpt_every": 1310720,
"eval_seeds": "0,1,2",
"bench_steps": 200000,
"runs_dir": "runs",
"run_name": "exp02_ppo_2x4",
"model_path": "",
"policy": "route"
}
```
## Provenance
- Code: https://github.com/BurnyCoder/4d-snake-rl at commit `212b78092dfcae4aa3947fb1106e7c65cc969c7c`.
- Library versions (`versions.json`): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
- `eval/summary.json` and `eval/eval_episodes.csv` are the files the repository's reports quote; every evaluated network is compared in [reports/networks.md](https://github.com/BurnyCoder/4d-snake-rl/blob/main/reports/networks.md).
- Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741
## Files
- `best_model.zip`: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
- `config.json`, `versions.json`: the run's resolved configuration and environment.
- `eval/`: evaluation summary and one row per evaluation episode.
- `train/progress.csv`: the SB3 training log; `figures/`: the learning curves.
## Licence
MIT, like the repository.
|