jepacpp/h-jepa-GGUF

H-JEPA — GGUF for jepa.cpp

H-JEPA (arXiv 2610.06805, Zhang, Terver, Rabbat, LeCun, Balestriero) trains a hierarchy of action-conditioned JEPA world models end to end: level 1 predicts one step ahead in the latent space of a ViT image encoder, each level above re-encodes every stride-th state of the level below and predicts one of its steps ahead, conditioned on a latent macro-action. These are the authors' released checkpoints (jepa-world-models/h-jepa, code at kevinghst/H-JEPA) converted to GGUF for jepa.cpp, a ggml C/C++ engine with no Python and no PyTorch at run time.

hjepa_l{N} is the N-level H-JEPA, hwm_l{N} the HWM baseline (identity upper encoders), lewm the flat one-level baseline. One seed of every released configuration is here (seed 42 for the simulated environments, seed 1 for DROID).

What runs

The encoder, every level's state encoder, action encoder, predictor and latent rollout, and the paper's top-down gradient planner (jepa_hjepa_* in include/jepa.h; jepa-plan goes from an observation and a goal image to actions).

hf download jepacpp/h-jepa-GGUF ant/ant_hjepa_l3-seed42-f16.gguf --local-dir models/gguf
jepa-plan -m models/gguf/ant/ant_hjepa_l3-seed42-f16.gguf --obs now.png --goal goal.png --horizons 2,2,5 --samples 100,100,400

Parity with the authors' PyTorch

Every file below was checked by test-predictor --hjepa against activations dumped from the authors' own modules (scripts/hjepa_reference.py): encoder tokens, each level's states, latent actions, action embeddings, teacher-forced predictions and autoregressive rollouts, and the whole chain from frames to top-level states. cos min is the worst row cosine over all of those comparisons, rel max the largest max-abs error relative to the reference's largest value.

file	levels	size	cos min	rel max
ant/ant_hjepa_l2-seed42-f32.gguf	2	94.4 MB	1.0000000	1.3e-06
ant/ant_hjepa_l2-seed42-f16.gguf	2	55.0 MB	0.9999910	3.1e-03
ant/ant_hjepa_l2-seed42-q8_0.gguf	2	36.6 MB	0.9933167	1.1e-01
ant/ant_hjepa_l3-seed42-f32.gguf	3	139.3 MB	1.0000000	7.7e-07
ant/ant_hjepa_l3-seed42-f16.gguf	3	81.9 MB	0.9999981	1.3e-03
ant/ant_hjepa_l3-seed42-q8_0.gguf	3	55.0 MB	0.9983362	4.8e-02
ant/ant_hjepa_l4-seed42-f32.gguf	4	184.2 MB	1.0000000	3.1e-06
ant/ant_hjepa_l4-seed42-f16.gguf	4	108.7 MB	0.9999975	2.4e-03
ant/ant_hwm_l2-seed42-f32.gguf	2	93.3 MB	1.0000000	7.0e-07
ant/ant_hwm_l2-seed42-f16.gguf	2	54.4 MB	0.9999997	8.7e-04
ant/ant_hwm_l2-seed42-q8_0.gguf	2	36.2 MB	0.9997584	2.2e-02
ant/ant_hwm_l3-seed42-f32.gguf	3	136.9 MB	1.0000000	3.1e-06
ant/ant_hwm_l3-seed42-f16.gguf	3	80.7 MB	0.9999977	2.2e-03
ant/ant_hwm_l3-seed42-q8_0.gguf	3	54.3 MB	0.9941525	7.7e-02
ant/ant_hwm_l4-seed42-f32.gguf	4	180.6 MB	1.0000000	9.9e-07
ant/ant_hwm_l4-seed42-f16.gguf	4	107.0 MB	0.9999956	3.3e-03
ant/ant_lewm-seed42-f32.gguf	1	49.6 MB	1.0000000	1.8e-05
ant/ant_lewm-seed42-f16.gguf	1	28.2 MB	0.9999982	1.3e-03
ant/ant_lewm-seed42-q8_0.gguf	1	18.1 MB	0.9984276	3.4e-02
cube/cube_hjepa_l2-seed42-f32.gguf	2	93.0 MB	1.0000000	8.9e-07
cube/cube_hjepa_l2-seed42-f16.gguf	2	52.2 MB	0.9999998	6.8e-04
cube/cube_hjepa_l2-seed42-q8_0.gguf	2	33.1 MB	0.9998490	1.5e-02
cube/cube_hjepa_l3-seed42-f32.gguf	3	140.8 MB	1.0000000	7.7e-07
cube/cube_hjepa_l3-seed42-f16.gguf	3	79.2 MB	0.9999998	7.2e-04
cube/cube_hjepa_l3-seed42-q8_0.gguf	3	50.4 MB	0.9997903	2.0e-02
cube/cube_hjepa_l4-seed42-f32.gguf	4	188.6 MB	1.0000000	1.3e-05
cube/cube_hjepa_l4-seed42-f16.gguf	4	106.3 MB	0.9999996	8.9e-04
cube/cube_hjepa_l4-seed42-q8_0.gguf	4	67.7 MB	0.9998970	1.5e-02
cube/cube_hwm_l2-seed42-f32.gguf	2	92.4 MB	1.0000000	9.3e-07
cube/cube_hwm_l2-seed42-f16.gguf	2	51.9 MB	0.9999998	5.1e-04
cube/cube_hwm_l2-seed42-q8_0.gguf	2	32.9 MB	0.9998793	1.3e-02
cube/cube_hwm_l3-seed42-f32.gguf	3	139.6 MB	1.0000000	9.2e-07
cube/cube_hwm_l3-seed42-f16.gguf	3	78.6 MB	0.9999998	5.3e-04
cube/cube_hwm_l3-seed42-q8_0.gguf	3	50.1 MB	0.9998652	2.1e-02
cube/cube_hwm_l4-seed42-f32.gguf	4	186.8 MB	1.0000000	7.3e-07
cube/cube_hwm_l4-seed42-f16.gguf	4	105.4 MB	0.9999996	1.0e-03
cube/cube_hwm_l4-seed42-q8_0.gguf	4	67.2 MB	0.9999145	1.3e-02
cube/cube_lewm-seed42-f32.gguf	1	45.2 MB	1.0000000	1.5e-06
cube/cube_lewm-seed42-f16.gguf	1	25.1 MB	0.9999998	5.9e-04
cube/cube_lewm-seed42-q8_0.gguf	1	15.7 MB	0.9998823	1.2e-02
fourroom/fourroom_hjepa_l2-seed42-f32.gguf	2	65.9 MB	1.0000000	4.9e-05
fourroom/fourroom_hjepa_l2-seed42-f16.gguf	2	37.7 MB	0.9996970	2.1e-02
fourroom/fourroom_hjepa_l2-seed42-q8_0.gguf	2	24.5 MB	0.9885078	1.1e-01
fourroom/fourroom_hjepa_l3-seed42-f32.gguf	3	86.7 MB	1.0000000	3.3e-05
fourroom/fourroom_hjepa_l3-seed42-f16.gguf	3	50.4 MB	0.9999836	4.4e-03
fourroom/fourroom_hjepa_l4-seed42-f32.gguf	4	107.6 MB	1.0000000	1.0e-04
fourroom/fourroom_hwm_l2-seed42-f32.gguf	2	65.3 MB	1.0000000	1.7e-05
fourroom/fourroom_hwm_l2-seed42-f16.gguf	2	37.4 MB	0.9999912	3.5e-03
fourroom/fourroom_hwm_l3-seed42-f32.gguf	3	85.6 MB	1.0000000	3.1e-05
fourroom/fourroom_hwm_l3-seed42-f16.gguf	3	49.8 MB	0.9999852	4.7e-03
fourroom/fourroom_hwm_l4-seed42-f32.gguf	4	105.8 MB	1.0000000	9.0e-05
fourroom/fourroom_hwm_l4-seed42-f16.gguf	4	62.1 MB	0.9999763	7.0e-03
fourroom/fourroom_lewm-seed42-f32.gguf	1	45.1 MB	1.0000000	6.1e-06
fourroom/fourroom_lewm-seed42-f16.gguf	1	25.1 MB	0.9999884	3.9e-03
fourroom/fourroom_lewm-seed42-q8_0.gguf	1	15.7 MB	0.9963070	5.8e-02
pusht/pusht_hjepa_l2-seed42-f32.gguf	2	92.9 MB	1.0000000	1.2e-06
pusht/pusht_hjepa_l2-seed42-f16.gguf	2	52.1 MB	0.9999992	7.2e-04
pusht/pusht_hjepa_l2-seed42-q8_0.gguf	2	33.0 MB	0.9994396	2.1e-02
pusht/pusht_hjepa_l3-seed42-f32.gguf	3	140.7 MB	1.0000000	1.5e-06
pusht/pusht_hjepa_l3-seed42-f16.gguf	3	79.2 MB	0.9999996	9.0e-04
pusht/pusht_hjepa_l3-seed42-q8_0.gguf	3	50.3 MB	0.9996304	2.3e-02
pusht/pusht_hjepa_l4-seed42-f32.gguf	4	188.5 MB	1.0000000	1.3e-06
pusht/pusht_hjepa_l4-seed42-f16.gguf	4	106.2 MB	0.9999997	7.1e-04
pusht/pusht_hjepa_l4-seed42-q8_0.gguf	4	67.7 MB	0.9997470	2.1e-02
pusht/pusht_hwm_l2-seed42-f32.gguf	2	92.3 MB	1.0000000	3.3e-06
pusht/pusht_hwm_l2-seed42-f16.gguf	2	51.8 MB	0.9999945	2.6e-03
pusht/pusht_hwm_l2-seed42-q8_0.gguf	2	32.8 MB	0.9982233	3.6e-02
pusht/pusht_hwm_l3-seed42-f32.gguf	3	139.5 MB	1.0000000	1.7e-05
pusht/pusht_hwm_l3-seed42-f16.gguf	3	78.6 MB	0.9997881	1.6e-02
pusht/pusht_hwm_l3-seed42-q8_0.gguf	3	50.0 MB	0.9969335	6.5e-02
pusht/pusht_hwm_l4-seed42-f32.gguf	4	186.7 MB	1.0000000	1.0e-06
pusht/pusht_hwm_l4-seed42-f16.gguf	4	105.3 MB	0.9999997	6.5e-04
pusht/pusht_hwm_l4-seed42-q8_0.gguf	4	67.2 MB	0.9987045	1.5e-02
pusht/pusht_lewm-seed42-f32.gguf	1	45.1 MB	1.0000000	8.5e-07
pusht/pusht_lewm-seed42-f16.gguf	1	25.1 MB	0.9999995	9.4e-04
pusht/pusht_lewm-seed42-q8_0.gguf	1	15.7 MB	0.9996572	2.1e-02
droid/droid_hjepa_l2-seed1-f32.gguf	2	345.8 MB	1.0000000	6.6e-07
droid/droid_hjepa_l2-seed1-f16.gguf	2	217.2 MB	0.9999999	4.2e-04
droid/droid_hjepa_l2-seed1-q8_0.gguf	2	156.9 MB	0.9997898	1.4e-02
droid/droid_hwm_l2-seed1-f32.gguf	2	343.4 MB	1.0000000	6.4e-07
droid/droid_hwm_l2-seed1-f16.gguf	2	216.0 MB	0.9999999	2.1e-04
droid/droid_hwm_l2-seed1-q8_0.gguf	2	156.3 MB	0.9997806	1.1e-02
droid/droid_lewm-seed1-f32.gguf	1	215.0 MB	1.0000000	9.2e-07
droid/droid_lewm-seed1-f16.gguf	1	130.1 MB	0.9999997	4.5e-04
droid/droid_lewm-seed1-q8_0.gguf	1	90.3 MB	0.9998426	1.2e-02

Measured at jepa.cpp 871b994, CPU. Bars: tests/test-predictor.cpp thresholds_for(): f32 cos >= 0.9999 and rel <= 1e-3; f16 cos_mean >= 0.9999, cos_min >= 0.99; q8_0 cos_mean >= 0.999, cos_min >= 0.98. q4_k and below do not clear the q8_0 bars on these small models and are not published.

The planner is checked against the authors' GradientSolver / HierarchicalSolver replayed on their own random draws: at f32 every cost, autograd gradient, solve and hierarchical plan matches, and at f16 every cost and gradient. A q8_0 file is verified as a forward model only; whether it plans as well as f32 has not been measured in closed loop.

Inputs

Frames are squash-resized to the encoder's image size and ImageNet-normalised (jepa.pre.*). Actions and proprioception are z-scored with the training statistics stored in the file (norm.action.*, norm.proprio.*, jepa_hjepa_norm_stats); a level-1 action is jepa.hjepa.frameskip consecutive raw actions stacked.

Licence

MIT, as the source checkpoints (jepa-world-models/h-jepa) and code. Cite the paper:

@article{zhang2026hjepa,
  title   = {H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning},
  author  = {Zhang, Wancong and Terver, Basile and Rabbat, Michael and LeCun, Yann and Balestriero, Randall},
  journal = {arXiv preprint arXiv:2610.06805},
  year    = {2026}
}