README.md

---
license: mit
library_name: stable-baselines3
pipeline_tag: reinforcement-learning
tags:
- 4d-snake
- 4d-snake-4x4
- snake
- reinforcement-learning
- deep-reinforcement-learning
- stable-baselines3
- sb3-contrib
- maskable-ppo
base_model: BurnyCoder/4d-snake-exp05-bc-4x4
model-index:
- name: 4d-snake-exp05b-ppo-4x4-from-bc
  results:
  - task:
      type: reinforcement-learning
      name: reinforcement-learning
    dataset:
      name: 4d-snake-4x4
      type: 4d-snake-4x4
    metrics:
    - type: completion_rate
      name: completion rate (deterministic, 100 episodes x 3 seeds)
      value: 1.000 +/- 0.000
      verified: false
    - type: mean_reward
      name: mean episode return (deterministic)
      value: 248.57
      verified: false
    - type: fill
      name: mean final fill (deterministic)
      value: 1.000
      verified: false
    - type: steps_to_complete
      name: mean steps to complete (deterministic, won episodes)
      value: 16413.6
      verified: false
---

# 4d-snake-exp05b-ppo-4x4-from-bc

An MLP policy with two hidden layers of 512 units for **4-dimensional snake on the 4^4 board** (256 cells, 8 moves): MaskablePPO fine-tuned from the behaviour-cloned network, no curriculum, 20,000,000 environment steps. From a length-1 start it completes the board in every deterministic evaluation episode, evaluated with the protocol of [docs/evaluation.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/docs/evaluation.md) (100 episodes x 3 seeds, masked `evaluate_policy`).

## Results (`eval/summary.json`)

| mode | completion +- std | mean fill | steps to complete | won within 4C |
|---|---|---|---|---|
| deterministic (argmax) | 1.000 +- 0.000 | 1.000 | 16,413.6 | 0.000 |
| sampling | 0.763 +- 0.017 | 0.884 | 16,641.6 | 0.000 |

## How to use

The observation is this repository's `4*C + 2` float vector and the action space its `2*ndim` masked moves ([docs/game_rules.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/docs/game_rules.md)), so the checkpoint runs inside `snake4d`'s environment:

```bash
git clone https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent.git && cd 4d-snake-reinforcement-learning-agent && uv sync
hf download BurnyCoder/4d-snake-exp05b-ppo-4x4-from-bc best_model.zip --local-dir weights
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=4 --set ndim=4
```

```python
# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
from sb3_contrib import MaskablePPO
from sb3_contrib.common.maskable.utils import get_action_masks
from snake4d.config import Config
from snake4d.vec_env import make_env

cfg = Config(size=4, ndim=4)
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
env = make_env(cfg, 1, 0)  # one board; observation shape (1, 4*C + 2)
obs = env.reset()
masks = get_action_masks(env)  # the legal moves, one row per board
action, _ = model.predict(obs, action_masks=masks, deterministic=True)
```

Use deterministic mode: the cloned policy follows a fixed Hamiltonian cycle and sampled off-cycle moves eventually trap the snake (see the results table; analysis: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/experiments/exp05_bc_4x4.md).

## Training

- Phase `train`; experiment file `experiments/exp05b_ppo_4x4_from_bc.env`; write-up: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/experiments/exp05_bc_4x4.md.
- Resolved configuration (`config.json`):

```json
{
  "size": 4,
  "ndim": 4,
  "idle_mult": 4,
  "r_food": 1.0,
  "r_death": -1.0,
  "r_win": 10.0,
  "r_step": -0.001,
  "shaping_coef": 0.0,
  "n_envs": 4096,
  "total_timesteps": 20000000,
  "n_steps": 64,
  "batch_size": 8192,
  "n_epochs": 4,
  "gamma": 0.99,
  "gae_lambda": 0.95,
  "lr_start": 0.0001,
  "lr_end": 1e-05,
  "clip_start": 0.1,
  "clip_end": 0.05,
  "ent_coef": 0.0,
  "vf_coef": 0.5,
  "max_grad_norm": 0.5,
  "target_kl": 0.03,
  "net_width": 512,
  "device": "auto",
  "torch_threads": 8,
  "seed": 0,
  "curriculum": 0,
  "curriculum_window": 4,
  "curriculum_delta": 0,
  "curriculum_rho": 0.9,
  "curriculum_min_eps": 500,
  "p_true_start": 0.2,
  "eval_episodes": 100,
  "eval_every": 2097152,
  "ckpt_every": 8388608,
  "eval_seeds": "0,1,2",
  "bench_steps": 200000,
  "bc_epochs": 20,
  "bc_lr": 0.001,
  "runs_dir": "runs",
  "run_name": "exp05b_ppo_4x4_from_bc",
  "model_path": "runs/20260904-221730_imitate_exp05_bc_4x4/bc_model.zip",
  "policy": "route"
}
```

![exp05b_ppo_4x4_from_bc_curves.png](figures/exp05b_ppo_4x4_from_bc_curves.png)

![exp05b_ppo_4x4_from_bc_fill_hist.png](figures/exp05b_ppo_4x4_from_bc_fill_hist.png)

## Provenance

- Code: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent at commit `016b5dc97583b7c086ad172226997e6aeea92dca`.
- Library versions (`versions.json`): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
- `eval/summary.json` and `eval/eval_episodes.csv` are the files the repository's reports quote; every evaluated network is compared in [reports/networks.md](https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/networks.md).
- Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741

## Files

- `best_model.zip`: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
- `config.json`, `versions.json`: the run's resolved configuration and environment.
- `eval/`: evaluation summary and one row per evaluation episode.
- `train/progress.csv`: the SB3 training log; `figures/`: the learning curves and the fill histogram.

## Licence

MIT, like the repository.