README.md

---
license: mit
tags:
  - scanpath-prediction
  - eye-tracking
  - saliency
  - computer-vision
---

# Scanpath Prediction Checkpoints

Model checkpoints for a transformer-based scanpath prediction architecture — predicting the
sequence of eye fixations `(x, y)` a human makes when free-viewing an image.

Evaluated against **DeepGaze III** (Kümmerer, Bethge & Wallis, *Journal of Vision* 2022) on
**MIT1003** and **CAT2000**.

---

## ⚠️ Which folder should I use?

**Use `v2_phase2/`.** It holds the current, published configuration. The `nb17/`, `nb19/`,
`nb20/` and `nb21_*` folders are *older exploratory runs* kept for provenance — they use
different hyperparameters (K=4 instead of K=8, plain L2 loss, learned-sigma variant) and
**will not reproduce the reported numbers**.

---

## Results (`v2_phase2`)

L2 = mean Euclidean distance between predicted and ground-truth fixation, with **each axis
normalized independently to [0,1]** (`x/W`, `y/H`). Teacher-forced, all valid steps 1..8,
10-fold cross-validation on predefined 8/1/1 splits (identical splits to DeepGaze III).

| Dataset | Ours | DeepGaze III | Improvement |
|---|---|---|---|
| **MIT1003** | **0.1227** | 0.2112 | **−42%** |
| **CAT2000** | **0.1432** | 0.1998 * | **−28%** |

\* DeepGaze III on CAT2000 is a **transfer** baseline — its scanpath components are
cross-validated on MIT1003 and were never retrained on CAT2000. We ensemble all 10 components.
Label it as transfer wherever the number is used.

### Per-step L2 (per-axis, [0,1])

| step | MIT1003 ours | MIT1003 DG3 | CAT2000 ours | CAT2000 DG3 |
|---|---|---|---|---|
| 1 | 0.1220 | 0.139 | 0.1051 | 0.168 |
| 2 | 0.1211 | 0.204 | 0.1176 | 0.175 |
| 3 | 0.1167 | 0.223 | 0.1389 | 0.172 |
| 4 | 0.1184 | 0.224 | 0.1510 | 0.176 |
| 5 | 0.1256 | 0.224 | 0.1576 | 0.184 |
| 6 | 0.1247 | 0.230 | 0.1583 | 0.187 |
| 7 | 0.1271 | 0.231 | 0.1591 | 0.189 |
| 8 | 0.1261 | 0.229 | 0.1606 | 0.188 |

Our model has lower error at every fixation step on both datasets.

---

## Contents

```
v2_phase2/
  mit1003/
    dinov2_big_unfrozen/     fold0..fold9  (+ trainsummary json)   best L2 on MIT1003
    clip_small_unfrozen/     fold0..fold9  (+ trainsummary json)   best NSS on MIT1003
  cat2000/
    dinov2_big_unfrozen/     fold0..fold9  (+ trainsummary json)

nb17_mit1003_cv10/               older baseline (pre-Phase-2), single-fold samples
nb19_mit1003_cv10_k4/            K_FIX=4 ablation
nb20_cat2000_cv10_k4/            K_FIX=4 ablation, CAT2000
nb21_mit1003_cv10_learned_sigma/ learned-sigma experiment
```

Each `v2_phase2` config ships **all 10 CV folds**, so the cross-validated numbers above are
fully reproducible. `trainsummary_*.json` records per-fold best val-NSS, per-epoch history
and wall-clock time.

---

## Architecture

```
image → encoder (frozen or fine-tuned) → transformer autoregressive decoder → Linear(d, 2) → (x, y)
```

| Config | Encoder | enc_dim | dec_dim | layers | batch |
|---|---|---|---|---|---|
| `dinov2_big` | DINOv2 ViT-B/14 | 512 | 512 | 10 | 32 |
| `clip_small` | CLIP | 512 | 256 | 4 | 128 |

- `K_FIX = 8` fixations predicted per image
- `context_len = 4` — the decoder's causal attention sees at most the last 4 fixations,
  matching DeepGaze III's history length
- First fixation is predicted from a BOS token at the image centre `(0.5, 0.5)`

## Training hyperparameters

```
image size    224 x 224          (448 was tested and was worse)
epochs        30
optimiser     AdamW (fused), weight_decay 5e-4
lr            3.3e-4 head; 2e-5 encoder when unfrozen  (differential LR)
schedule      3-epoch linear warmup -> cosine
precision     bfloat16 AMP, drop_last=True
dropout       0.1
grad clip     max_norm 1.0
augmentation  horizontal flip (train only), x -> 1-x
loss          masked_L2 + 0.3 * NSS_aux  (map_size 128, sigmas 3/5/8, active from epoch 3)
selection     best checkpoint by validation NSS (not validation L2)
seed          42
CV            10 folds, 8/1/1, predefined splits (identical to DeepGaze III)
```

## Usage

```python
import torch
from huggingface_hub import hf_hub_download
from scanpath_lib.models.scanpath_model import ScanpathModel

path = hf_hub_download(
    "businesslion/scanpath-prediction-checkpoints",
    "v2_phase2/mit1003/dinov2_big_unfrozen/fold0_dinov2_big_unfrozen.pt",
)

model = ScanpathModel(
    k_fix=8, enc_dim=512, dec_dim=512, n_heads=4, n_layers=10,
    max_len=64, encoder_name="dinov2", image_size=224,
    context_len=4, predict_sigma=False,
)
model.load_state_dict(torch.load(path, map_location="cpu", weights_only=True))
model.eval()
```

For the cross-validated numbers, evaluate each fold's checkpoint on **that fold's test split**
(every image is then scored exactly once by a model that never trained on it) and pool.

## Reproducibility note

The `v2_phase2` weights were retrained in Aug 2026 after the originals were lost, using the
same code, seed and fold splits but a newer PyTorch (2.10). They reproduce the original
numbers to within **0.7%** (MIT1003 0.1219 → 0.1227; CAT2000 0.1426 → 0.1432). Exact
bit-level equality is not expected across PyTorch versions.

## Citation

DeepGaze III baseline:

```bibtex
@article{kummerer2022deepgaze3,
  title   = {DeepGaze III: Modeling free-viewing human scanpaths with deep learning},
  author  = {K{\"u}mmerer, Matthias and Bethge, Matthias and Wallis, Thomas S. A.},
  journal = {Journal of Vision},
  volume  = {22}, number = {5}, pages = {7}, year = {2022},
  doi     = {10.1167/jov.22.5.7}
}
```