---
base_model:
- facebook/ijepa_vith16_1k
datasets:
- dralois/Bar-JEPA
language:
- en
library_name: pytorch
license: cc-by-nc-4.0
metrics:
- f1
- accuracy
- precision
- recall
pipeline_tag: keypoint-detection
tags:
- bar-chart
- chart-understanding
- document-ai
- ijepa
- vit
- synthetic
- keypoint-detection
- self-supervised
model-index:
- name: Bar-JEPA (kp-cl-arp-ctt-ft)
  results:
  - task:
      type: keypoint-detection
      name: Bar Chart Value Extraction
    dataset:
      name: UB PMC (real-world)
      type: real-world
    metrics:
    - type: f1
      value: 0.785
      name: Bar Keypoint F1
    - type: f1
      value: 0.842
      name: Tick Keypoint F1
    - type: accuracy
      value: 0.499
      name: Value Accuracy (ε=0.05)
    - type: accuracy
      value: 0.365
      name: Value Accuracy (ε=0.02)
  - task:
      type: keypoint-detection
      name: Bar Chart Value Extraction
    dataset:
      name: Synthetic (100 held-out charts)
      type: dralois/Bar-JEPA
    metrics:
    - type: f1
      value: 0.961
      name: Bar Keypoint F1
    - type: f1
      value: 0.951
      name: Tick Keypoint F1
    - type: accuracy
      value: 0.792
      name: Value Accuracy (ε=0.05)
    - type: accuracy
      value: 0.657
      name: Value Accuracy (ε=0.02)
---

# Bar-JEPA

Per-bar numerical value recovery from vertical bar chart images. A self-supervised I-JEPA encoder (ViT-H, finetuned on synthetic bar charts) produces feature maps consumed by a lightweight keypoint decoder. The decoder outputs heatmaps for bar corners, value-axis ticks and the coordinate origin, which are post-processed with NMS, OCR and RANSAC regression to recover numerical bar values.

**Paper:** [*Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture*](https://huggingface.co/papers/2608.06062) — Poonam, Epple & Ropinski, Ulm University (ICDAR 2026).  
**Code:** [github.com/dralois/Bar-JEPA](https://github.com/dralois/Bar-JEPA)

## Pipeline

```
Image → variable-resolution patch extraction (≤256 patches, 14 px)
      → frozen I-JEPA ViT-H encoder
      → classic decoder (2× deconv-BN-ReLU + 1×1 conv, 32 keypoint channels)
      → origin / classification / regression heatmap heads
      → NMS + PaddleOCR + RANSAC → bar values
```

The encoder is frozen during decoder training. Variable-resolution inputs follow the Pix2Struct aspect-ratio-preserving scaling strategy.

## Checkpoints

Download checkpoints and place them in `./output/`. The ViT-H base checkpoint (`IN1K-vit.h.14-300e.pth.tar`) must also be present there before encoder finetuning.

| File | Description |
|---|---|
| `kp-cl-arp-ctt-ft-latest.pth.tar` | Classic decoder, ARP encoder + Chart-to-Text FT, UB PMC finetuned (**best**) |
| `kp-cl-arp-ft-latest.pth.tar` | Classic decoder, ARP encoder, UB PMC finetuned |
| `kp-cl-noarp-ft-latest.pth.tar` | Classic decoder, fixed-resolution encoder, UB PMC finetuned |
| `kp-cl-vanilla-ft-latest.pth.tar` | Classic decoder, vanilla ImageNet-only encoder, UB PMC finetuned |
| `kp-spl-arp-ft-latest.pth.tar` | Simple decoder, ARP encoder, UB PMC finetuned |

## Usage

All tasks go through `bar-jepa/main.py`. Requires [pixi](https://pixi.sh) or PyTorch ≥ 2.3 + PaddleOCR.

```bash
# Setup
pixi install
# or: pip install -e ".[torch]" && pip install paddlepaddle paddleocr
```

**Encoder finetuning:**
```bash
python bar-jepa/main.py \
  --mode finetune \
  --fname bar-jepa/configs/charts/vith14_arp.yaml \
  --devices cuda:0
```

**Decoder training (pretraining → UB PMC finetuning):**
```bash
python bar-jepa/main.py --mode decoder \
  --fname bar-jepa/configs/keypoint/classic_arp.yaml --devices cuda:0

python bar-jepa/main.py --mode decoder \
  --fname bar-jepa/configs/keypoint/classic_arp.yaml --devices cuda:0 \
  --override meta.do_finetune=true data.root_path=./UBPMC data.is_ubpmc=true
```

**Evaluation:**
```bash
python bar-jepa/main.py --mode eval \
  --fname bar-jepa/configs/eval/classic_arp.yaml --devices cuda:0

# Run all five configurations at once:
python scripts/run_all_evals.py
```

## Training

**Encoder finetuning** — self-supervised I-JEPA objective on 100k synthetic bar charts. 50 epochs, AdamW, cosine LR 5×10⁻⁵ → 1×10⁻⁶ (6-epoch warm-up), cosine weight decay 0.02–0.04, effective batch size 40, 2× NVIDIA RTX A6000. Optional additional 25 epochs on 15k Chart-to-Text real-world charts.

**Decoder training** — supervised keypoint regression on 17k synthetic charts. 50 epochs, AdamW, cosine LR 1×10⁻³ → 1×10⁻⁵ (3-epoch warm-up), cosine weight decay 0.04–0.1, batch size 1360, 1× NVIDIA RTX A6000. Finetuned for 30 more epochs on UB PMC / ICPR CHART-Infographics 2022 (1316 vertical bar charts).

## Performance

Variable-resolution encoder (ARP), classic decoder, synthetic pretraining only:

| Dataset | Bar F1 | Tick F1 | Acc (ε=0.05) | Acc (ε=0.02) |
|---|---|---|---|---|
| UB PMC (real-world) | 0.740 | 0.827 | 0.450 | 0.341 |
| Synthetic (100 charts) | 0.956 | 0.940 | 0.778 | 0.629 |

With additional Chart-to-Text real-world finetuning:

| Dataset | Bar F1 | Tick F1 | Acc (ε=0.05) | Acc (ε=0.02) |
|---|---|---|---|---|
| UB PMC (real-world) | 0.785 | 0.842 | 0.499 | 0.365 |
| Synthetic (100 charts) | 0.961 | 0.951 | 0.792 | 0.657 |

Accuracy criterion: `|h_gt − h_pred| / h_gt ≤ ε` (same as Zhou et al. 2021).

## Dataset

Training data at [`dralois/Bar-JEPA`](https://huggingface.co/datasets/dralois/Bar-JEPA). Download and place at `./data` (encoder, 100k) and `./data_decoder` (decoder, 17k), or generate from scratch:

```bash
python bar-gen/generator.py --output ./data --count 100000
```

| Config | Split | Samples | Local path |
|---|---|---|---|
| `encoder_training` | train / test | 100,000 / 100 | `./data` |
| `decoder_training` | train / test | 17,000 / 3,400 | `./data_decoder` |
| `pipeline_testing` | test | 100 | — |

## Limitations

- Vertical bar charts only; no stacked bars, error bars or 3D effects.
- OCR (PaddleOCR `latin_PP-OCRv5_mobile_rec`) is an external dependency for tick label reading.
- Encoder operates in latent space only, making integration with multimodal language models non-trivial.

## Citation

```bibtex
@inproceedings{poonam2026bar-jepa,
  title     = {Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture},
  author    = {Poonam, Poonam and Epple, Alexander and Ropinski, Timo},
  booktitle = {ICDAR},
  year      = {2026}
}
```

## License

Model code is derived from [facebookresearch/ijepa](https://github.com/facebookresearch/ijepa) and licensed under the same terms. See [bar-jepa/LICENSE](https://github.com/dralois/Bar-JEPA/blob/main/bar-jepa/LICENSE).