File size: 5,676 Bytes
0104112 d40e768 0104112 a668434 0104112 a668434 0104112 a668434 0104112 a668434 0104112 a668434 0104112 d40e768 0104112 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | ---
license: mit
library_name: stable-baselines3
pipeline_tag: reinforcement-learning
tags:
- 4d-snake
- 4d-snake-3x4
- snake
- reinforcement-learning
- deep-reinforcement-learning
- stable-baselines3
- sb3-contrib
- maskable-ppo
- backplay-curriculum
model-index:
- name: 4d-snake-exp03c-ppo-3x4-backplay-relaxed
results:
- task:
type: reinforcement-learning
name: reinforcement-learning
dataset:
name: 4d-snake-3x4
type: 4d-snake-3x4
metrics:
- type: completion_rate
name: completion rate (deterministic, 100 episodes x 3 seeds)
value: 0.000 +/- 0.000
verified: false
- type: mean_reward
name: mean episode return (deterministic)
value: 41.75
verified: false
- type: fill
name: mean final fill (deterministic)
value: 0.542
verified: false
---
# 4d-snake-exp03c-ppo-3x4-backplay-relaxed
An MLP policy with two hidden layers of 512 units for **4-dimensional snake on the 3^4 board** (81 cells, 8 moves): MaskablePPO trained from scratch, Backplay reverse curriculum (gate 0.8, window 8), 30,000,000 environment steps. From a length-1 start it completes the board in 0.0 % of deterministic episodes, evaluated with the protocol of [docs/evaluation.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/docs/evaluation.md) (100 episodes x 3 seeds, masked `evaluate_policy`).
**Negative result.** This network never fills the board from the true start (mean final fill 0.542). It is published so the failure is reproducible; the write-up linked below analyses why.
## Results (`eval/summary.json`)
| mode | completion +- std | mean fill | steps to complete | won within 4C |
|---|---|---|---|---|
| deterministic (argmax) | 0.000 +- 0.000 | 0.542 | never | 0.000 |
| sampling | 0.000 +- 0.000 | 0.545 | never | 0.000 |
## How to use
The observation is this repository's `4*C + 2` float vector and the action space its `2*ndim` masked moves ([docs/game_rules.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/docs/game_rules.md)), so the checkpoint runs inside `snake4d`'s environment:
```bash
git clone https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent.git && cd 4d-snake-reinforcement-learning-agent && uv sync
hf download BurnyCoder/4d-snake-exp03c-ppo-3x4-backplay-relaxed best_model.zip --local-dir weights
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=3 --set ndim=4
```
```python
# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
from sb3_contrib import MaskablePPO
from sb3_contrib.common.maskable.utils import get_action_masks
from snake4d.config import Config
from snake4d.vec_env import make_env
cfg = Config(size=3, ndim=4)
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
env = make_env(cfg, 1, 0) # one board; observation shape (1, 4*C + 2)
obs = env.reset()
masks = get_action_masks(env) # the legal moves, one row per board
action, _ = model.predict(obs, action_masks=masks, deterministic=True)
```
## Training
- Phase `train`; experiment file `experiments/exp03c_ppo_3x4_backplay_relaxed.env`; write-up: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/experiments/exp03_ppo_3x4.md.
- Resolved configuration (`config.json`):
```json
{
"size": 3,
"ndim": 4,
"idle_mult": 4,
"r_food": 1.0,
"r_death": -1.0,
"r_win": 10.0,
"r_step": -0.001,
"shaping_coef": 0.0,
"n_envs": 2048,
"total_timesteps": 30000000,
"n_steps": 64,
"batch_size": 8192,
"n_epochs": 4,
"gamma": 0.99,
"gae_lambda": 0.95,
"lr_start": 0.0003,
"lr_end": 1e-05,
"clip_start": 0.2,
"clip_end": 0.05,
"ent_coef": 0.01,
"vf_coef": 0.5,
"max_grad_norm": 0.5,
"target_kl": 0.03,
"net_width": 512,
"device": "auto",
"torch_threads": 8,
"seed": 0,
"curriculum": 1,
"curriculum_window": 8,
"curriculum_delta": 4,
"curriculum_rho": 0.8,
"curriculum_min_eps": 500,
"p_true_start": 0.2,
"eval_episodes": 100,
"eval_every": 1310720,
"ckpt_every": 5242880,
"eval_seeds": "0,1,2",
"bench_steps": 200000,
"runs_dir": "runs",
"run_name": "exp03c_ppo_3x4_backplay_relaxed",
"model_path": "",
"policy": "route"
}
```
## Provenance
- Code: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent at commit `9dfaa8a6f662d57fc8a02ee67a15efa3cbefab6c`.
- Library versions (`versions.json`): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
- `eval/summary.json` and `eval/eval_episodes.csv` are the files the repository's reports quote; every evaluated network is compared in [reports/networks.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/networks.md).
- Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741
## Files
- `best_model.zip`: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
- `config.json`, `versions.json`: the run's resolved configuration and environment.
- `eval/`: evaluation summary and one row per evaluation episode.
- `train/progress.csv`: the SB3 training log; `figures/`: the learning curves and the fill histogram.
## Licence
MIT, like the repository.
|