File size: 5,114 Bytes
0215d42 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | ---
base_model: datalab-to/chandra-ocr-2
pipeline_tag: image-text-to-text
library_name: llama.cpp
license: other
license_name: ai-pubs-open-rail-m-modified
license_link: https://huggingface.co/datalab-to/chandra-ocr-2/blob/af93b47dba1b47b6640c86ccf487ed2260ab9a09/LICENSE
tags:
- gguf
- bf16
- ocr
- vision-language
- llama.cpp
- rocm
- amd
- rdna4
- gfx1201
---
# Chandra OCR 2 BF16 GGUF — one-GPU RDNA 4 validation
Unofficial, reproducible BF16 GGUF conversion of
[`datalab-to/chandra-ocr-2`](https://huggingface.co/datalab-to/chandra-ocr-2), validated end to end with
[`llama.cpp`](https://github.com/ggml-org/llama.cpp) on one AMD Radeon RX 9070 XT (`gfx1201`).
This is a container conversion, not a fine-tune, new model, or claim of improved OCR quality. The main GGUF preserves
BF16 precision while omitting Chandra's unused multi-token-prediction (MTP) draft head. The vision projector is BF16.
The original model, architecture, training, and authorship belong to Datalab.
## Files
| File | Bytes | SHA-256 |
|---|---:|---|
| `chandra-ocr-2.BF16.gguf` | 9,695,791,648 | `4e9d5fa9854cf820d4425d28034df31ec1221a7f9d1082b0c4359d79f318cb56` |
| `chandra-ocr-2.mmproj-bf16.gguf` | 675,568,864 | `54ddb8285933512cdbf1c84238aa0435b473a6efef2caeda8ca802c2899e87b3` |
| `chat_template.jinja` | 7,622 | `0d158f349ca965f7eea9db0eb45cd177b85bb0e4ae05dcdd0f060da8f7d41812` |
The complete machine-readable provenance is in [`manifest.json`](manifest.json).
## Pinned provenance
- Source model: `datalab-to/chandra-ocr-2`
- Source revision: `af93b47dba1b47b6640c86ccf487ed2260ab9a09`
- Source `model.safetensors` SHA-256: `0804568be9f099d6479fad9ed77a4da4611f3c1e7bc6e009af7dce45e8aa3847`
- Converter/runtime: `ggml-org/llama.cpp`
- Converter revision: `8f5ab832ca7d8a7b4f23687693fb8b0ecbc227e7`
- `chandra-ocr`: `0.2.0`
See [`CONVERSION.md`](CONVERSION.md) for the exact commands and integrity checks.
## Verified configuration
| Component | Verified value |
|---|---|
| GPU | AMD Radeon RX 9070 XT, 16 GB (`gfx1201`) |
| GPU allocation | One isolated GPU; full layer offload |
| OS / kernel | Ubuntu 24.04.4 / 6.17.0-40-generic |
| ROCm | 7.2.1 |
| llama.cpp | `8f5ab832ca7d8a7b4f23687693fb8b0ecbc227e7` |
| Context | 24,576 tokens |
| Maximum tested output envelope | 12,384 tokens |
| Observed model-process VRAM peak | Approximately 11.2 GB |
The GGUF format is not RDNA4-specific. `gfx1201` is the hardware on which this exact pair completed image, PDF,
financial-table, handwritten-document, deterministic-output, and lifecycle tests. Other llama.cpp-supported hardware may
work but is not validated here.
## Run with llama.cpp and the Chandra CLI
Build llama.cpp for your accelerator, then start a loopback server. These are the validated inference settings; replace
the device selector as appropriate for your machine.
```bash
ROCR_VISIBLE_DEVICES=GPU-YOUR-STABLE-UUID HIP_VISIBLE_DEVICES=0 \
llama-server \
--model chandra-ocr-2.BF16.gguf \
--mmproj chandra-ocr-2.mmproj-bf16.gguf \
--alias chandra \
--host 127.0.0.1 \
--port 18100 \
--ctx-size 24576 \
--n-gpu-layers 999 \
--split-mode none \
--main-gpu 0 \
--flash-attn on \
--fit off \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 512 \
--jinja \
--chat-template-file chat_template.jinja \
--image-min-tokens 1024
```
In another shell:
```bash
python3 -m venv .venv
.venv/bin/pip install 'chandra-ocr==0.2.0'
VLLM_API_BASE=http://127.0.0.1:18100/v1 \
VLLM_API_KEY=EMPTY \
VLLM_MODEL_NAME=chandra \
.venv/bin/python -m chandra.scripts.cli input.pdf output \
--method vllm \
--batch-size 1 \
--max-workers 1 \
--max-retries 2 \
--max-output-tokens 12384 \
--no-images \
--no-html
```
Do not expose an unauthenticated llama.cpp server to a public network. Generative OCR can omit or hallucinate content;
verify consequential documents against their source.
## Validation result and limits
The exact-pinned conversion matched the public comparison BF16 tensor schema. All projector tensor payloads matched. In
the main file, every tensor payload matched except twelve scalar F32 values across ten `ssm_a` tensors; the maximum
absolute difference was `9.5367431640625e-07`. Accepted OCR outputs matched. There is no evidence that this conversion
improves OCR quality over other correct BF16 conversions.
The useful contribution is the pinned, MTP-free artifact pair; complete provenance; one-16-GB-GPU validation; and a
reproducible launch envelope.
## License and attribution
Chandra OCR 2 weights use Datalab's **AI Pubs Open RAIL-M License (Modified)**. It contains usage, redistribution,
commercial, competitive-use, attribution, and share-alike conditions. Read [`LICENSE`](LICENSE) in full before using or
redistributing these files. The license and its restrictions apply to this derivative conversion.
Modified-file notice: the original Chandra OCR 2 checkpoint was converted to GGUF at the pinned llama.cpp revision; the
unused MTP draft head was omitted from the main GGUF; model tensor precision otherwise remains BF16. No Datalab
endorsement is claimed.
|