--- license: mit library_name: jepa.cpp tags: - jepa - ggml - gguf - jepa.cpp - v-jepa - v-jepa-2 - world-model - robotics - video - planning --- # V-JEPA 2-AC ViT-g (action-conditioned world model) — GGUF for jepa.cpp Meta's **V-JEPA 2-AC** — the action-conditioned world model behind the paper's zero-shot robot planning — converted to GGUF for [jepa.cpp](https://github.com/aselimc/jepa.cpp), a ggml C/C++ engine that runs it on a plain CPU with no Python and no PyTorch. One bundle: the frozen ViT-g/16 encoder from `vjepa2-ac-vitg.pt` plus the 24-layer, 1024-dim predictor that takes a 7-d end-effector action and a 7-d pose per frame and predicts the next frame's latents, block-causally over frames. `jepa_ac_rollout` scores K candidate action sequences in one graph per step, and `jepa_ac_energy` is the L1 planning energy a CEM planner minimises. **Note on the encoder:** it is *not* `facebook/vjepa2-vitg-fpc64-256`. Meta's AC checkpoint carries its own frozen ViT-g, which agrees with the HF release only to cosine ~0.998 per tensor; `encoder` and `target_encoder` inside it are bit-identical. This bundle ships the checkpoint's own, which is what `vjepa2_ac_vit_giant` loads. **1317 M** parameters; D = 1408, 40 layers, 22 heads, patch 16, tubelet 2, 256x256. Everything the engine needs — dimensions, positional scheme, preprocessing recipe, and class labels where there are any — travels inside the file, so inference needs one binary and one GGUF and nothing else. ## Run it ```bash git clone --recursive https://github.com/aselimc/jepa.cpp && cd jepa.cpp cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build build -j hf download jepacpp/vjepa2-ac-vitg-GGUF vjepa2-ac-vitg-f16.gguf --local-dir models/gguf # encode a frame, roll 4 candidate action sequences out 2 steps, score them against a goal build/jepa-worldmodel --ac -m vjepa2-ac-vitg-f16.gguf --image ctx.png --goal goal.png \ --actions-npy actions.npy # float32 [K, H, 7] ``` `--pool` selects `mean`, `cls`, `lewm` or `none` (the full token map); `-o` writes a `.npy`. `scripts/download_models.sh` fetches whole sets at once. The C API is one header, [`include/jepa.h`](https://github.com/aselimc/jepa.cpp/blob/main/include/jepa.h) — full reference on the [C API page](https://aselimc.github.io/jepa.cpp/api/). ## Files | file | size | sha256 (first 16) | tier | measured against the PyTorch reference | |---|---|---|---|---| | `vjepa2-ac-vitg-f32.gguf` | 5025.5 MiB | `234b7e775da2ff28` | exact | — | | `vjepa2-ac-vitg-f16.gguf` | 2519.0 MiB | `d57cc1760f90f779` | parity | — | | `vjepa2-ac-vitg-q8_0.gguf` | 1344.1 MiB | `3dffe7130d7f7efd` | parity, **below the bar** | — | | `vjepa2-ac-vitg-q4_0.gguf` | 717.4 MiB | `554f3a9f293a2b80` | advisory, **below the bar** | — | | `vjepa2-ac-vitg-q4_k.gguf` | 717.4 MiB | `0d2fb4e022e2d1ca` | advisory, **below the bar** | — | ᵖ `tests/test-parity` on the CPU backend, stored reference input, 32 threads, worst sample — [docs/parity.md](https://aselimc.github.io/jepa.cpp/parity/). ᵈ `scripts/gguf_dequant_selftest.py`: the dequantized weights through the numpy reference graph at f32 activations, so the figure is the weight error alone — [docs/quantization.md](https://aselimc.github.io/jepa.cpp/quantization/). `cos mean` is the mean per-token cosine of `last_hidden_state`, `worst` its single worst token. **Tiers.** `exact` — reproduces the PyTorch reference to the printed precision on the CPU. `parity` — passes its family's `test-parity` thresholds. `advisory` — below 8 bits per weight, which is not a parity configuration: the results are reported, only the derived tensors and the top-1 are gated. Which file to ship: [Accuracy → which dtype](https://aselimc.github.io/jepa.cpp/accuracy/#which-dtype-to-ship). `vjepa2-ac-vitg-q8_0.gguf` multi-step rollouts. `vjepa2-ac-vitg-q4_0.gguf` multi-step rollouts. `vjepa2-ac-vitg-q4_k.gguf` multi-step rollouts. Full checksums: ``` 234b7e775da2ff28e29d6b12a48fd28f6b0e888b1854581a684dc8e8e84848bb vjepa2-ac-vitg-f32.gguf d57cc1760f90f779eba12d90d04b5028334a95f9b6960da6a98239b9a6f2441c vjepa2-ac-vitg-f16.gguf 3dffe7130d7f7efd0805d2d714e72a232632077cc59a9036b7896435f6c40ef6 vjepa2-ac-vitg-q8_0.gguf 554f3a9f293a2b80d3e0f9dd40f033e4ca1a291640712606a727a419ee0f4425 vjepa2-ac-vitg-q4_0.gguf 0d2fb4e022e2d1ca9a3aaf93d5ae3ad51b1d41244b04f634561ba2ac48a5dfbc vjepa2-ac-vitg-q4_k.gguf ``` Verify a download with `sha256sum -c`. The other types `jepa-quantize` can produce (`q4_1`, `q5_0`, `q5_1`, `q5_k`, `q6_k`, measured in [quantization](https://aselimc.github.io/jepa.cpp/quantization/)) are not published here; make them locally with `build/jepa-quantize vjepa2-ac-vitg-f16.gguf out.gguf q6_k -t 32`. ## Measured Every figure below is read from a committed artifact of jepa.cpp [`00bfd4e`](https://github.com/aselimc/jepa.cpp/commit/00bfd4e) by `scripts/hf_publish.py` — [parity](https://aselimc.github.io/jepa.cpp/parity/), [quantization](https://aselimc.github.io/jepa.cpp/quantization/), [accuracy](https://aselimc.github.io/jepa.cpp/accuracy/), [performance](https://aselimc.github.io/jepa.cpp/performance/) and `tests/results/*.json`. **Speed** — the encoder graph at f16 on 32 threads (AMD Ryzen Threadripper PRO 7995WX 96-Cores): 2228 ms per 16-frame clip; 15747 ms per 64-frame clip. The same shape on NVIDIA RTX 4500 Ada Generation: 27.8 ms. Peak RSS at f16: 3092 MiB. **Plan with f16.** A rollout compounds — step h's input is step h-1's output — so the worst predicted token of a 2-step rollout falls from 0.9925 at f16 to 0.9368 at q8_0 to 0.5429 at q4_k, and at q4_k on a GPU the planning energy misranks the candidates and the model picks a different action. The encoder half of the bundle passes at every tier on both backends. At f32 on the CPU the predictor is exact to cosine 1.0000000 against Meta's own world-model dump, and K candidates batched on the graph's batch axis are bit-identical to K sequential rollouts, and `jepa_ac_plan` reproduces Meta's own CEM planner to 3e-08 on its own recorded random draws. Full tables in [parity](https://aselimc.github.io/jepa.cpp/parity/). **On `-q4_k.gguf`:** K-quants need `ne[0] % 256 == 0`, and the encoder's width is 1408 = 5.5 x 256, so every `[*, 1408]`-row matrix falls back to q4_0. In THIS bundle the 24 predictor blocks are 1024-wide and keep the K-quant, so the mix is 121 q4_0 against 137 q4_K and the file reports `general.file_type = 12 (q4_k)`. (The encoder-only `vjepa2-vitg-fpc64-256-q4_k.gguf` goes the other way: 158 q4_0 / 52 q4_K, reported as q4_0.) `jepa-info` lists the real mix. ## Source, licence and attribution Converted from [`dl.fbaipublicfiles.com/vjepa2/vjepa2-ac-vitg.pt`](https://dl.fbaipublicfiles.com/vjepa2/vjepa2-ac-vitg.pt). **MIT.** The checkpoint is published by **Meta AI (FAIR)** at `https://dl.fbaipublicfiles.com/vjepa2/vjepa2-ac-vitg.pt`; the licence is the [LICENSE](https://github.com/facebookresearch/vjepa2/blob/main/LICENSE) of `facebookresearch/vjepa2` (Copyright (c) Meta Platforms, Inc. and affiliates). No gating, no acceptable-use policy. These GGUF files are the same weights re-serialised, quantized where the file name says so. Cite the [V-JEPA 2 paper](https://arxiv.org/abs/2506.09985). The licence travels inside every GGUF as `general.license` and the origin as `general.source_url`; `build/jepa-info --kv` prints them. jepa.cpp's own code is MIT. ## Conversion Produced by jepa.cpp [`00bfd4e`](https://github.com/aselimc/jepa.cpp/commit/00bfd4e): ```bash scripts/download_models.sh --convert vjepa2-ac python scripts/convert.py --family vjepa2_ac --src models/vjepa2_ac/vjepa2-ac-vitg.pt --out models/gguf/vjepa2-ac-vitg-f16.gguf --ftype f16 # ... and again with --ftype f32 --out models/gguf/vjepa2-ac-vitg-f32.gguf for the f32 file for q in q8_0 q4_0 q4_k; do build/jepa-quantize models/gguf/vjepa2-ac-vitg-f16.gguf \ models/gguf/vjepa2-ac-vitg-$q.gguf $q -t 32 done ``` `jepa-quantize` re-types only the 2-D attention / FFN / projection / classifier matrices; patch embeddings, position tables, norms and biases keep the source type. The rules are in [docs/gguf-schema.md](https://aselimc.github.io/jepa.cpp/gguf-schema/). ## Links - Code: - Documentation: - Parity fixtures: - All jepa.cpp GGUFs: