README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
library_name: transformers
datasets:
  - comma-project/deep-jsonl
language:
  - la
pipeline_tag: image-text-to-text
tags:
  - ocr
  - htr
  - handwritten-text-recognition
  - manuscripts
  - palaeography
  - catmus
  - medieval-latin
  - qwen3.5
  - vision-language
metrics:
  - cer
  - wer
---

# comma-qwen-3.5-0.8b-full-33k

`Qwen/Qwen3.5-0.8B` fine-tuned to produce **CATMuS-compliant, line-by-line transcriptions**
of medieval Latin manuscript pages.

Given a page image, the model returns one line of text for each physical written
line, in reading order. CATMuS is a *graphemic* standard: the sequence of letters
and signs as written, reduced to the modern Latin alphabet, with no editorial
intervention. The output is not a translation and not a normalised reading.

This is a **full fine-tune** — a standalone checkpoint, no adapter and no `peft` dependency at inference.

**This is a continued fine-tune.** It starts from `comma-qwen-3.5-0.8b-full (9,906 pages)` and trains further on new pages, so the *base* column below is that earlier model rather than a stock checkpoint — the gain shown is what the additional data bought, measured within one job.

- **Training data** — 33,112 pages from [`comma-project/deep-jsonl`](https://huggingface.co/datasets/comma-project/deep-jsonl)
- **Evaluation** — 243 held-out pages, each a different hand, none seen in training
- **Demo** — [comma-qwen-3.5-demo](https://huggingface.co/spaces/wjbmattingly/comma-qwen-3.5-demo)

## Results

Greedy decoding, 3072 max new tokens, 2048 visual tokens per page,
batch 4. "Base" is the previous **comma-qwen-3.5-0.8b-full (9,906 pages)**, scored on the same pages in the same job — a base number from another job on another
day is not a control.

| metric | base | **this model** |
|---|---|---|
| CER (NFD) | 0.1458 | **0.1240** |
| CER (raw code points) | 0.1859 | 0.1651 |
| WER (NFD) | 0.4029 | 0.3584 |
| line recall (NFD) | 0.1672 | 0.1883 |
| CER macro (NFD) | 0.1318 | 0.1385 |
| degenerate pages | 0.0041 | 0.0000 |
| truncated pages | 0.0165 | 0.0123 |

CER/WER are micro-averaged (total edits ÷ total reference characters), so a long
page outweighs a short one; the macro mean is given alongside because a gap
between them says the errors are concentrated. `line_recall` is the share of
reference lines reproduced exactly and in order — for a line-by-line CATMuS
target that is the number a palaeographer looks at first.

## Usage

```python
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

REPO = "wjbmattingly/comma-qwen-3.5-0.8b-full-33k"
BASE = "Qwen/Qwen3.5-0.8B"

processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForImageTextToText.from_pretrained(
    REPO, dtype=torch.bfloat16, device_map="cuda"
)
model.eval()

# The prompt is the 11.6k-character CATMuS rule set the model was trained under.
# Serving a different prompt serves a different task: the rules are what the
# target obeys, so the model was taught to read them as part of the input.
prompt = open("prompt.txt", encoding="utf-8").read()

image = Image.open("page.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [{"type": "image"}, {"type": "text", "text": prompt}],
}]

# `enable_thinking=False` is not optional. With no kwarg, Qwen3.5 0.8B/2B leave
# the thinking block CLOSED and 4B/9B leave it OPEN -- and a model trained on
# pure transcription that is handed an open block will think instead of
# transcribing, emitting prose until the token budget runs out.
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
assert text.count("<think>") == text.count("</think>"), "thinking block left open"

inputs = processor(text=[text], images=[[image]], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(
        **inputs,
        do_sample=False,             # greedy is the published control
        repetition_penalty=1.1,      # the measured serving default; 1.0 reproduces the table
        max_new_tokens=3072,
        eos_token_id=[processor.tokenizer.eos_token_id],
        pad_token_id=processor.tokenizer.pad_token_id,
    )

print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:],
                                 skip_special_tokens=True).strip())
```

`prompt.txt` is in the
[demo Space](https://huggingface.co/spaces/wjbmattingly/comma-qwen-3.5-demo/blob/main/prompt.txt).

### The image budget travels with the checkpoint

This repo's `processor_config.json` is already capped to **2048 visual
tokens** per page (2,097,152 pixels after smart-resize, one token per
32×32 block), which is what training and scoring used. Load the processor from
this repo rather than from the base model: the library default is 16,777,216
pixels — 16k visual tokens for one page — and serving at that budget shows the
model a page at a resolution it never saw.

## Training

| | |
|---|---|
| tuning | full fine-tune |
| trainable parameters | 852,985,920 of 852,985,920 (100.0%) |
| epochs | 2 |
| effective batch | 8 |
| learning rate | 5e-05 |
| max sequence length | 8192 |
| visual tokens per page | 2048 |
| precision | bf16 |
| hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |
| wall time | 38.9 h |

`max_length` never truncates: every sample is measured first and over-budget
ones are dropped, because a truncated completion teaches the model to stop
mid-transcription and a truncated prompt removes the rules the target obeys.

## Limitations

- **Nothing under 0.02 CER is a result.** Four seeds of one configuration in this
  project gave CER 0.1271 / 0.1334 / 0.1370 / 0.1485 — mean 0.1365, sd 0.0090.
  bf16 training is nondeterministic across nodes and greedy decoding amplifies a
  sub-millivolt logit difference into a different token. Treat differences
  smaller than ~0.02 as ties.
- **Greedy is deterministic given identical batching, but not batch-size
  invariant.** Re-scoring the same pages at batch 1 instead of 4 changes almost
  every prediction and moves aggregate CER by ±0.013 for the smallest model here.
  Any comparison across a batch-size change is invalid.
- **Quote `cer_nfd`, not `cer`.** `ũ` (U+0169) and `u` + combining tilde are one
  glyph, and 77% of training targets mix the two forms — so an edit distance over
  raw code points charges two edits for a difference that is not on the page,
  about 16% of all edit operations. Every checkpoint is biased toward one form
  and training does not fix it (the targets are mixed, so there is no signal to
  fit), which means raw `cer` partly scores which Unicode form a tokenizer
  prefers. NFD and not NFC: `t̃ r̃ m̃ p̃ c̃ q̃` have no precomposed form.
- **The remaining error is mostly convention, not reading.** On the best run of
  this project, word-division spaces are ~17% of all edit operations and
  allographs another ~5%, against ~1% for genuine letter confusion (`r`↔`s`,
  `f`↔`s`). Both are decisions about the transcription convention. Do not read a
  CER at this level as a statement about how well the model reads the script.
- **Latin, and this convention.** Trained on Latin-dominant manuscripts under the
  CATMuS graphemic standard. deep-jsonl uses the full MUFI superscript-letter
  repertoire (`qͥ` qui, `qͣ` qua, `qͦ` quo) where coarser conventions flatten almost
  everything to a tilde, so scoring this model against a corpus transcribed more
  coarsely costs 1–2% CER for reasons unrelated to reading.
- **Blank and non-text pages** return the literal string `VLM-NO-TEXT`.

## Citation

```bibtex
@misc{comma_qwen35,
  title  = {comma-qwen-3.5: CATMuS transcription models for medieval Latin manuscripts},
  author = {Mattingly, William J. B.},
  year   = {2026},
  url    = {https://huggingface.co/wjbmattingly/comma-qwen-3.5-0.8b-full-33k}
}
```