README.md

---
license: mit
base_model: dots-studio/dots.mocr
base_model_relation: finetune
library_name: transformers
pipeline_tag: image-text-to-text
language:
- ko
- zh
tags:
- ocr
- document-parse
- layout
- vertical-text
- historical-documents
- hanja
- classical-chinese
- custom_code
- lora
metrics:
- cer
---

# Bandit OCR

**Built with dots.mocr.**

Bandit OCR reads a page of a Korean historical document and returns one region per printed column,
in reading order, with the characters of that column. It is a LoRA fine-tune of
[dots.mocr](https://huggingface.co/dots-studio/dots.mocr), a 1.7B-parameter document vision-language model,
trained on 229,356 annotated pages of Joseon-dynasty woodblock prints and manuscripts.

On the held-out test split it reads a page at 0.0195 character error rate with a column
F1 of 0.9856, against 0.1436 for the AI Hub ResNet pipeline and
0.2398 for NDLkotenOCR on the same pages.

The repository holds both forms of the same fine-tune. At the root are the merged weights, the
adapter already folded in at scale 0.75, which load and serve exactly like the base and need
no PEFT. Under `lora/` is the LoRA adapter itself, for composing onto the base yourself,
changing the scale or training further. Take the root unless you know you want the adapter.

- Desktop application and project site: https://bandit.dhchoi.net
- Paper: JADH 2026, September 2026

## Model details

| | |
|---|---|
| Base model | [dots-studio/dots.mocr](https://huggingface.co/dots-studio/dots.mocr) at revision `e539fbb52280393adc081b289ec597430a0f9031` |
| Parameters | 1.7B (1.2B language decoder + 0.4B vision tower) |
| Adaptation | LoRA, rank 64, alpha 128, dropout 0.05, on the language projections and the vision tower |
| Weights | root: the adapter folded in at scale 0.75, a plain bfloat16 checkpoint, no PEFT needed |
| Adapter | `lora/`: the same LoRA unmerged, 580 MB, to apply at scale 0.75 yourself |
| Files | 2 safetensors shards at the root, about 6.1 GB, plus the adapter |
| Checkpoint | t3_full step 56000 of the full training run |
| Trained on | 229,356 annotated pages of Korean historical documents |
| Input | one page image, 3,136 to 11,289,600 pixels after `smart_resize` |
| Output | JSON array of column regions in reading order |
| Author | Donghyeok Choi, Department of History, Hong Kong Baptist University |
| Project | https://bandit.dhchoi.net |

## What it returns

One JSON array, one object per printed column, in reading order:

```json
[
  {"bbox": [2205, 354, 2350, 3406], "category": "Text", "text": "李敷金輅孫興宗沈孝生高呂張至和咸傅霖韓尙敬黃居正"},
  {"bbox": [2057, 366, 2193, 3430], "category": "Text", "text": "任彦忠張思靖閔汝翼等大小臣僚及閑良耆老等奉國寶詣"}
]
```

Those are the first two of the fifteen columns of one Sillok page from the test split, 2400 x 3744
pixels, on its 2408 x 3752 input grid.

Reading order is the convention of the printed page: column slots right to left, each slot top to
bottom, and the two sub-columns of an interlinear note (세주) right to left. An interlinear note is
a column region like any other, at about half width, with no markup in its text.

`bbox` is `[x1, y1, x2, y2]` in the resized input grid, not in original pixels. Map it back with
the same `smart_resize` the processor used:

```python
from qwen_vl_utils.vision_process import smart_resize

grid_h, grid_w = smart_resize(height, width, min_pixels=3136, max_pixels=11289600)
x1 = round(x1 * width / grid_w)
y1 = round(y1 * height / grid_h)
```

## Usage

### Transformers

The model ships the base's remote code, so `trust_remote_code=True` is required. It was trained and
evaluated against `transformers==4.51.3`; 4.52 and later change the Qwen2.5-VL processor contract
the custom `DotsVLProcessor` was written for.

```python
import json

import torch
from PIL import Image
from qwen_vl_utils import process_vision_info
from qwen_vl_utils.vision_process import smart_resize
from transformers import AutoModelForCausalLM, AutoProcessor

MODEL = "dhchoi/bandit-ocr"
PROMPT = (
    "Please output the layout information from the PDF image, including each layout element's "
    "bbox, its category, and the corresponding text content within the bbox.\n\n"
    "1. Bbox format: [x1, y1, x2, y2]\n\n"
    "2. Layout Categories: The possible categories are ['Caption', 'Footnote', 'Formula', "
    "'List-item', 'Page-footer', 'Page-header', 'Picture', 'Section-header', 'Table', 'Text', "
    "'Title'].\n\n"
    "3. Text Extraction & Formatting Rules:\n"
    "    - Picture: For the 'Picture' category, the text field should be omitted.\n"
    "    - Formula: Format its text as LaTeX.\n"
    "    - Table: Format its text as HTML.\n"
    "    - All Others (Text, Title, etc.): Format their text as Markdown.\n\n"
    "4. Constraints:\n"
    "    - The output text must be the original text from the image, with no translation.\n"
    "    - All layout elements must be sorted according to human reading order.\n\n"
    "5. Final Output: The entire output must be a single JSON object.\n"
)
MIN_PIXELS, MAX_PIXELS = 3136, 11289600

model = AutoModelForCausalLM.from_pretrained(
    MODEL, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)

path = "page.jpg"
with Image.open(path) as im:
    width, height = im.size
grid_h, grid_w = smart_resize(height, width, min_pixels=MIN_PIXELS, max_pixels=MAX_PIXELS)

messages = [{"role": "user", "content": [
    {"type": "image", "image": path, "min_pixels": MIN_PIXELS, "max_pixels": MAX_PIXELS},
    {"type": "text", "text": PROMPT},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
images, videos = process_vision_info(messages)
images = [im.resize((grid_w, grid_h), Image.Resampling.LANCZOS) for im in images]
inputs = processor(text=[text], images=images, videos=videos, padding=True,
                   return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(
        **inputs,
        max_new_tokens=8192,
        do_sample=False,
        eos_token_id=processor.tokenizer.convert_tokens_to_ids("<|endofassistant|>"),
        suppress_tokens=[151643, 151672],
    )
answer = processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
regions = json.loads(answer)
```

### vLLM

vLLM 0.21.0 serves the merged weights with no conversion; it resolves `DotsOCRForCausalLM` from the
remote code itself.

```bash
vllm serve dhchoi/bandit-ocr \
  --trust-remote-code --dtype bfloat16 --max-model-len 24576 \
  --limit-mm-per-prompt '{"image": 1}' \
  --mm-processor-kwargs '{"max_pixels": 11289600, "min_pixels": 3136}'
```

Send the image as a data URL with the prompt above, `temperature` 0, `max_tokens` 8192, and the two
decode settings of the section below: `"stop_token_ids": [151673]` and
`"logit_bias": {"151643": -100, "151672": -100}`.

### The LoRA adapter

`lora/` holds the same fine-tune unmerged, for composing onto the base yourself, changing
the scale or training further. The delta is scaled by 0.75 before it is applied: that scale is
part of the released model, not a training artefact, and every number below was measured with it.

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(
    "dots-studio/dots.mocr", revision="e539fbb52280393adc081b289ec597430a0f9031",
    trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(base, "dhchoi/bandit-ocr", subfolder="lora")

for module in model.modules():                  # every LoRA layer scales its delta by scaling[name]
    scaling = getattr(module, "scaling", None)
    if isinstance(scaling, dict):
        for name in scaling:
            scaling[name] *= 0.75

model = model.merge_and_unload()
```

That reproduces the weights at the root of this repository. Prompt, decode and rescale boxes the
same way, and set the stop id yourself, because the composed model inherits the base's generation
config.

Applied at its trained scale of 1.0 the adapter is a different, measurably worse model: on the same
540-page validation set an earlier checkpoint of this run read at 0.0656 page CER at scale 1.0
against 0.0376 at scale 0.75, both under the stop rule of the day.

## Decoding: stop on one token, suppress two

This matters more than any other setting here. The training target ends with `<|endofassistant|>`
(151673) and nothing else, but the base generation config also treats `<|endoftext|>` (151643) and
`<|assistant|>` (151672) as stop ids. The fine-tune sometimes emits `<|endoftext|>` in the middle of
a column at a hard glyph, and a page that stops there is truncated or fails to parse.

- `generation_config.json` at the root of this repository already sets `eos_token_id` to 151673
  alone. If you compose the adapter onto the base yourself, set it there: the base's config stops
  on all three.
- Suppress 151643 and 151672 during decoding as well (`suppress_tokens` in `generate`, a -100
  `logit_bias` through an OpenAI-compatible server).

Measured on 540 validation pages, this rule takes the released checkpoint from 0.0517 page CER with
2 pages failing to parse down to 0.0271 with none, weights unchanged. On the 16 pages of a
diagnostic cohort that stopped early at a neighbouring checkpoint, micro CER fell from 0.652 to
0.027 while six control pages stayed put. Every number reported below was measured under this rule.

## Evaluation

The whole held-out test split of the training corpus, 14,376 pages read through vLLM with
the decode rule above. Regions are matched one to one at IoU 0.5; CER is over matched
regions (region CER) and over the concatenated reading-order text of the page (page CER).
The layout row scores every test page that has ground-truth geometry; the text row scores
the pages that also carry ground-truth text.

| | Pages | P | R | F1 | mean IoU | region CER | page CER |
|---|---:|---:|---:|---:|---:|---:|---:|
| layout ground truth | 13,636 | 0.9897 | 0.9805 | 0.9851 | 0.9580 | 0.0256 | 0.0235 |
| text ground truth | 12,935 | 0.9909 | 0.9804 | 0.9856 | 0.9583 | 0.0217 | 0.0195 |

By script style, on the text ground truth:

| | Pages | P | R | F1 | mean IoU | region CER | page CER |
|---|---:|---:|---:|---:|---:|---:|---:|
| haeseo 해서 (standard) | 5,573 | 0.9894 | 0.9713 | 0.9803 | 0.9488 | 0.0309 | 0.0252 |
| haengseo 행서 (semi-cursive) | 1,558 | 0.9692 | 0.9779 | 0.9735 | 0.9447 | 0.0484 | 0.0634 |
| choseo 초서 (cursive) | 88 | 0.9555 | 0.8274 | 0.8869 | 0.9112 | 0.2272 | 0.2063 |
| yeseo 예서 (clerical) | 15 | 0.9508 | 1.0000 | 0.9748 | 0.9475 | 0.0412 | 0.0648 |
| sillok (print, no style label) | 5,701 | 0.9966 | 0.9896 | 0.9931 | 0.9681 | 0.0125 | 0.0101 |

By collection:

| | Pages | P | R | F1 | mean IoU | region CER | page CER |
|---|---:|---:|---:|---:|---:|---:|---:|
| AI Hub 234 (고서 한자 인식) | 2,578 | 0.9738 | 0.9801 | 0.9769 | 0.9485 | 0.0312 | 0.0404 |
| AI Hub 603 (고서 한자 인식 OCR) | 2,391 | 0.9874 | 0.9533 | 0.9700 | 0.9414 | 0.0428 | 0.0331 |
| AI Hub 71294 (OCR 데이터 고서한자) | 2,265 | 0.9934 | 0.9810 | 0.9872 | 0.9536 | 0.0374 | 0.0335 |
| Sillok 조선왕조실록 | 5,701 | 0.9966 | 0.9896 | 0.9931 | 0.9681 | 0.0125 | 0.0101 |

### Against the baselines, same pages, same ground truth

| System | layout F1 | page CER |
|---|---:|---:|
| Bandit OCR (this model) | 0.9851 | 0.0195 |
| AI Hub ResNet pipeline | 0.9595 | 0.1436 |
| NDLkotenOCR | 0.8880 | 0.2398 |
| dots.ocr, base | 0.1602 | 0.1497 |

The base dots.ocr layout F1 is a unit mismatch rather than a reading failure: the base
answers with about 3 regions where the ground truth has 15, so its page CER is the
comparable figure. A base dots.mocr row is not measured yet.

### Out of domain

1,000 pages of Chinese historical print (HisDoc1B), a tradition the model never saw:

| | Pages | P | R | F1 | mean IoU | region CER | page CER |
|---|---:|---:|---:|---:|---:|---:|---:|
| HisDoc1B | 1,000 | 0.6773 | 0.9512 | 0.7912 | 0.9165 | 0.2144 | 0.2241 |

Recall 0.95 says the columns are found and page CER 0.22 is the transfer number. The
precision is over-segmentation: the model returns 28,983 regions against the ground
truth's 20,638, and a quarter of its regions hold five characters or fewer.

## Training data

229,356 pages and 3,023,103 column regions, split off a corpus of four Korean
collections by book volume so that no volume straddles train and test (seed 20260828, 90/5/5):

- 조선왕조실록 Veritable Records of the Joseon Dynasty, National Institute of Korean History:
  99,764 training pages. Column boxes come from a character detector run over each page, and the
  characters of each column come from aligning the article's transcription against that detection.
- AI Hub 고서 한자 인식 datasets 234, 603 and 71294: 129,592 training pages of woodblock prints and
  manuscripts in five script styles, with character-level boxes grouped into columns.

The corpus itself is not published and cannot be: the AI Hub data may not be redistributed and the
Sillok images are copyrighted by the National Institute of Korean History. Only the weights are
released.

The text is the printed form: Hanja, the `○` article marker, `□` for a glyph with no code point,
and note text. Editorial punctuation, bracket marks and collation insertions are removed.

## Training procedure

One run on H200 GPUs: bfloat16, gradient checkpointing, effective batch 8 (per-device batch 1 with
gradient accumulation), AdamW, cosine schedule, warmup ratio 0.03, weight decay 0.01, seed 3407,
`flash_attention_2`, one page per sample at up to 11,289,600 pixels, batches grouped by length. The learning rate was 2e-4 for the first 32,000 steps and 5e-5 from there, on a
cosine schedule spanning the remaining steps; step 56000 is about 1.95 epochs over the
229,356-page training split.

Checkpoints were evaluated every 4,000 steps by generating a 540-page validation set and scoring it
the way the test split is scored, and the released checkpoint is the best of that curve by page CER.
The curve is flat from step 44,000 on, within 0.0036 CER of the selected point.

The LoRA delta is applied at scale 0.75. That is not a training setting: scaling the merged
delta below 1.0 was measured to read better than the trained scale, and it is folded into the merged
weights because a server applies no runtime scale.

## Limitations

- Cursive (초서) is the weak style: page CER 0.2063 against 0.0252 on standard script, and the
  lowest recall in the corpus (0.8274). The test split holds only 88 cursive pages, so that number
  is itself uncertain.
- The clerical (예서) row is 15 pages. Read it as an indication, not a score.
- The Sillok layout score is not independent. The v3 Sillok column boxes were drawn by
  HRCenterNet, the AI Hub baseline's own detector, and this model was trained on them. A 100-page
  human-corrected gold set confirms the column inventory (304 boxes drawn blind against 304), so
  recall and F1 on Sillok are meaningful, but the mean IoU on Sillok pages carries that circularity.
- Out of domain it over-segments. On Chinese material it splits small text, headings and notes
  into several regions where the ground truth keeps one column, which costs precision, not reading.
- It is a column reader, not a general document parser. The prompt is the base's full layout
  prompt, but the fine-tune answers with `Text` column regions; tables, formulas and figures are not
  part of the training target and the base's behaviour on them has been trained away.
- No safety or content filtering of any kind. It transcribes what is on the page.

## A note on the remote code

The custom modelling files are the base's, with four idempotent patches this project needs on its
hosts, all of them to how attention is chosen and imported, none of them to the architecture or the
weights:

- the flash-attn import is wrapped in a try/except, so the file imports on a host without it;
- `DotsVisionTransformer` declares `_supports_sdpa` and `_supports_flash_attn`, which transformers
  checks before it will give a sub-model sdpa;
- the sdpa path skips its all-true attention mask when the batch holds a single image, which is
  exact and avoids a mask that costs about 95 GB on a full page;
- `DotsVLProcessor` passes `video_processor` to the Qwen2.5-VL processor by keyword, which
  transformers 4.51 otherwise reads as `chat_template`.

`config.json` asks the vision tower for `flash_attention_2`. Without flash-attn installed it falls
back to eager attention, which is quadratic in the number of patches; set
`vision_config.attn_implementation` to `sdpa` in that case.

## Licence and attribution

Built with dots.mocr.

This model is a fine-tune of [dots.mocr](https://huggingface.co/dots-studio/dots.mocr), copyright
Xingyin Information Technology (Shanghai) Co., Ltd, used under the dots.mocr LICENSE AGREEMENT. A
copy of that agreement and the base `NOTICE` ship in this repository (clauses 7.1 and 7.2), the
notices in them are unchanged, and this card displays the attribution statement clause 7.3
requires. Nothing here implies endorsement by the licensor, and no name in this project is a
licensor trade name (clause 4.4). The weights are released on the terms the base is released
under; where the agreement and the metadata tag differ, the agreement is the document that
ships.

The training data is licensed separately and is not redistributed; see the section above.

## Citation

```bibtex
@inproceedings{choi2026bandit,
  title     = {Bandit: A Human-in-the-Loop OCR System for East Asian Historical Documents
               with Vision-Language Model Fine-tuning},
  author    = {Choi, Donghyeok},
  booktitle = {Japanese Association for Digital Humanities (JADH) 2026},
  year      = {2026}
}
```