README.md

---
license: mit
library_name: stable-baselines3
pipeline_tag: reinforcement-learning
tags:
- 4d-snake
- 4d-snake-3x4
- snake
- reinforcement-learning
- deep-reinforcement-learning
- stable-baselines3
- sb3-contrib
- maskable-ppo
- backplay-curriculum
model-index:
- name: 4d-snake-exp03c-ppo-3x4-backplay-relaxed
  results:
  - task:
      type: reinforcement-learning
      name: reinforcement-learning
    dataset:
      name: 4d-snake-3x4
      type: 4d-snake-3x4
    metrics:
    - type: completion_rate
      name: completion rate (deterministic, 100 episodes x 3 seeds)
      value: 0.000 +/- 0.000
      verified: false
    - type: mean_reward
      name: mean episode return (deterministic)
      value: 41.75
      verified: false
    - type: fill
      name: mean final fill (deterministic)
      value: 0.542
      verified: false
---

# 4d-snake-exp03c-ppo-3x4-backplay-relaxed

An MLP policy with two hidden layers of 512 units for **4-dimensional snake on the 3^4 board** (81 cells, 8 moves): MaskablePPO trained from scratch, Backplay reverse curriculum (gate 0.8, window 8), 30,000,000 environment steps. From a length-1 start it completes the board in 0.0 % of deterministic episodes, evaluated with the protocol of [docs/evaluation.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/docs/evaluation.md) (100 episodes x 3 seeds, masked `evaluate_policy`).

**Negative result.** This network never fills the board from the true start (mean final fill 0.542). It is published so the failure is reproducible; the write-up linked below analyses why.

## Results (`eval/summary.json`)

| mode | completion +- std | mean fill | steps to complete | won within 4C |
|---|---|---|---|---|
| deterministic (argmax) | 0.000 +- 0.000 | 0.542 | never | 0.000 |
| sampling | 0.000 +- 0.000 | 0.545 | never | 0.000 |

## How to use

The observation is this repository's `4*C + 2` float vector and the action space its `2*ndim` masked moves ([docs/game_rules.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/docs/game_rules.md)), so the checkpoint runs inside `snake4d`'s environment:

```bash
git clone https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent.git && cd 4d-snake-reinforcement-learning-agent && uv sync
hf download BurnyCoder/4d-snake-exp03c-ppo-3x4-backplay-relaxed best_model.zip --local-dir weights
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=3 --set ndim=4
```

```python
# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
from sb3_contrib import MaskablePPO
from sb3_contrib.common.maskable.utils import get_action_masks
from snake4d.config import Config
from snake4d.vec_env import make_env

cfg = Config(size=3, ndim=4)
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
env = make_env(cfg, 1, 0)  # one board; observation shape (1, 4*C + 2)
obs = env.reset()
masks = get_action_masks(env)  # the legal moves, one row per board
action, _ = model.predict(obs, action_masks=masks, deterministic=True)
```

## Training

- Phase `train`; experiment file `experiments/exp03c_ppo_3x4_backplay_relaxed.env`; write-up: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/experiments/exp03_ppo_3x4.md.
- Resolved configuration (`config.json`):

```json
{
  "size": 3,
  "ndim": 4,
  "idle_mult": 4,
  "r_food": 1.0,
  "r_death": -1.0,
  "r_win": 10.0,
  "r_step": -0.001,
  "shaping_coef": 0.0,
  "n_envs": 2048,
  "total_timesteps": 30000000,
  "n_steps": 64,
  "batch_size": 8192,
  "n_epochs": 4,
  "gamma": 0.99,
  "gae_lambda": 0.95,
  "lr_start": 0.0003,
  "lr_end": 1e-05,
  "clip_start": 0.2,
  "clip_end": 0.05,
  "ent_coef": 0.01,
  "vf_coef": 0.5,
  "max_grad_norm": 0.5,
  "target_kl": 0.03,
  "net_width": 512,
  "device": "auto",
  "torch_threads": 8,
  "seed": 0,
  "curriculum": 1,
  "curriculum_window": 8,
  "curriculum_delta": 4,
  "curriculum_rho": 0.8,
  "curriculum_min_eps": 500,
  "p_true_start": 0.2,
  "eval_episodes": 100,
  "eval_every": 1310720,
  "ckpt_every": 5242880,
  "eval_seeds": "0,1,2",
  "bench_steps": 200000,
  "runs_dir": "runs",
  "run_name": "exp03c_ppo_3x4_backplay_relaxed",
  "model_path": "",
  "policy": "route"
}
```

![exp03c_ppo_3x4_backplay_relaxed_curves.png](figures/exp03c_ppo_3x4_backplay_relaxed_curves.png)

![exp03c_ppo_3x4_backplay_relaxed_fill_hist.png](figures/exp03c_ppo_3x4_backplay_relaxed_fill_hist.png)

## Provenance

- Code: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent at commit `9dfaa8a6f662d57fc8a02ee67a15efa3cbefab6c`.
- Library versions (`versions.json`): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
- `eval/summary.json` and `eval/eval_episodes.csv` are the files the repository's reports quote; every evaluated network is compared in [reports/networks.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/networks.md).
- Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741

## Files

- `best_model.zip`: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
- `config.json`, `versions.json`: the run's resolved configuration and environment.
- `eval/`: evaluation summary and one row per evaluation episode.
- `train/progress.csv`: the SB3 training log; `figures/`: the learning curves and the fill histogram.

## Licence

MIT, like the repository.