institutional-newspapers.md

metadata

name: institutional-newspapers
description: >-
  Use the Institutional Newspapers family of Hugging Face assets from the
  Institutional Data Initiative: historical newspaper scans segmented into
  individual crops, each enriched with OCR (Tesseract and VLM), crop-type
  classification, reading order, language detection, text metrics, named
  entities, subjects, and precomputed text and image embeddings. Currently
  covers the Boston Public Library collection (1.47M scans, 1795-1930), plus the
  three models that produced it and the production pipeline. Use when a task
  involves this corpus, reading or analyzing historical newspaper text or
  layout, reassembling an article from a page, finding advertisements or
  illustrations, tracking entities or subjects across newspapers, segmenting or
  classifying a new newspaper scan, or running the pipeline on another
  collection.

Institutional Newspapers - dataset, models & pipeline

Institutional Newspapers is a growing family of Hugging Face assets from the Institutional Data Initiative: each instance takes one institution's historical newspaper collection, segments every scan into individual type-agnostic crops, and enriches each crop with OCR and analysis. New collections are added over time. Browse the Institutional Newspapers collection to see what is available; a given instance may publish a different set of repos.

This skill covers the instance published so far - the Boston Public Library collection:

Repo	Type	Contents
institutional/institutional-newspapers-bpl	dataset	One row per scan. 1,473,635 scans, 83,147,041 crops
institutional/institutional-newspapers-segmenter-yolo26x	model	Scan -> crop bounding boxes
institutional/institutional-newspapers-crop-classifier-image-yolo26m-cls	model	Crop image -> crop type (7 classes)
institutional/institutional-newspapers-crop-classifier-text-model2vec	model	Crop text -> crop type (6 classes)
institutional-newspapers-pipeline	code	The 15-step pipeline that produced the dataset

Scale: 1,473,635 public domain scans published between 1795 and 1930, 83,147,041 crops, 16.3 billion o200k_base tokens of VLM OCR text and 14.7 billion from Tesseract.

Two ideas that drive every decision:

1. The crop_* columns are parallel lists that share one index, already in reading order. One row is one scan. Every crop_* column holds one entry per crop, and all of them are sorted by the reading order the pipeline detected for that page. zip them to reassemble a crop. Filtering preserves reading order, because it only drops entries from lists that are already sorted.
2. The dataset is precomputed pipeline output. Read the field instead of re-running the model. All three models were already run over the whole collection and their results stored as columns. Only run a model on a new scan or crop that is not in the dataset.

Pick the right asset

I need to...	Use
Crops, OCR, and all enrichment for a scan in the collection	dataset institutional-newspapers-bpl (large - stream it)
Metadata only, without scan images, OCR text, or embeddings	Parquet column projection (see below)
The crop type of a crop (already computed)	field crop_classification_gen
The per-modality signal behind that decision	fields crop_classification_image_only_gen / crop_classification_text_only_gen (+ _conf_gen)
The reading order of a page (already computed)	the order of the crop_* lists themselves
Entities or subjects for a crop (already computed)	crop_ner_per_gen / crop_ner_loc_gen / crop_ner_org_gen, crop_subject_gen (experimental)
Vector search over crops	crop_text_embeddings (256-d) / crop_image_embeddings (384-d)
Find crops on a new newspaper scan	model ...-segmenter-yolo26x
Classify a new crop	models ...-classifier-image-... + ...-classifier-text-..., then the combination rule
Process a new collection end to end	the pipeline

Dataset

institutional-newspapers-bpl

One row per newspaper scan (one page of one issue). A single train split. Large, because each row embeds its scan image - always stream unless you project columns. See Auth & gating.

from datasets import load_dataset

ds = load_dataset(
    "institutional/institutional-newspapers-bpl",
    split="train",
    streaming=True,
    token=True,  # gated repo: uses HF_TOKEN / cached login
)

row = next(iter(ds))
print(row["issue_id_src"], row["page_number_gen"], row["year_ext"])
print(row["scan_image"])  # a PIL image, decoded from the embedded WEBP bytes

# The crop_* columns share one index and are already in reading order.
for bbox, category, text in zip(
    row["crop_bbox_gen"],
    row["crop_classification_gen"],
    row["crop_vlm_ocr_gen"],
):
    # One value per crop, NULL where the step produced no result.
    print(category or "", bbox, text or "")

Reassemble the readable text of a page by keeping only the crop types you want. The result stays in reading order:

KEEP = {"Content", "Section heading"}

article = "\n\n".join(
    text
    for category, text in zip(row["crop_classification_gen"], row["crop_vlm_ocr_gen"])
    if category and category[0] in KEEP and text
)

Loading only specific columns

Read metadata from the Parquet shards directly with column projection. Parquet is columnar, so the bytes of the skipped columns are never transferred. This is the difference between a small download and a very large one.

import pyarrow.parquet as pq
from huggingface_hub import HfFileSystem

SKIP = {
    "scan_image",
    "crop_tesseract_ocr_gen",
    "crop_text_embeddings",
    "crop_image_embeddings",
}

fs = HfFileSystem(token=True)
shards = fs.glob("datasets/institutional/institutional-newspapers-bpl/**/*.parquet")

with fs.open(shards[0], cache_type="none") as handle:
    parquet_file = pq.ParquetFile(handle, pre_buffer=True)
    keep = [
        field.name
        for field in parquet_file.schema_arrow
        if field.name not in SKIP
    ]
    table = parquet_file.read(columns=keep)

print(table.num_rows, table.column_names)

Shards are named BPL-part-NNNNN.parquet and hold up to 250 scans each. They are compressed with zstd and written with a row-group size of 10.

Models

All under institutional/. Run them in order on a new scan; for scans already in the collection, read the dataset instead.

...-newspapers-segmenter-yolo26x (ultralytics)

A YOLO26x fine-tune. Single-class object detection: a scan in, one bounding box per crop out, regardless of crop type. Precision 0.927, recall 0.910, F1 0.918, mAP50 0.955, mAP50-95 0.901 on 153 held-out scans (2,991 bboxes).

from huggingface_hub import snapshot_download
from ultralytics import YOLO

model_path = snapshot_download("institutional/institutional-newspapers-segmenter-yolo26x")
model = YOLO(f"{model_path}/best.pt")

results = model("newspaper_scan.jpg", imgsz=960, iou=0.15, conf=0.6)
boxes = [b.xyxy.tolist()[0] for b in results[0].boxes]  # crop these from the scan

Use imgsz=960, iou=0.15, conf=0.6. The confidence threshold drops low-certainty detections and the deliberately low IoU threshold suppresses heavily overlapping boxes, which is what these dense layouts need.

...-crop-classifier-image-yolo26m-cls (ultralytics)

A YOLO26m-cls fine-tune. A cropped image in, one of the seven crop types out. Top-1 accuracy 0.913. Much stronger than the text classifier on the visual categories: F1 0.84 on Photograph or illustration against 0.48, and 0.92 on Cartoon against 0.68.

model_path = snapshot_download("institutional/institutional-newspapers-crop-classifier-image-yolo26m-cls")
model = YOLO(f"{model_path}/best.pt")

r = model("crop.jpg", imgsz=768)[0]
print(r.names[r.probs.top1], float(r.probs.top1conf))

Use imgsz=768.

...-crop-classifier-text-model2vec (model2vec)

A Model2Vec fine-tune of minishlab/potion-base-32M with a classifier head - light and CPU-friendly. The OCR text of a crop in, one of six crop types out. It does not learn Empty. Accuracy 0.92.

from model2vec.inference import StaticModelPipeline

model = StaticModelPipeline.from_pretrained(
    "institutional/institutional-newspapers-crop-classifier-text-model2vec"
)
print(model.predict(["CLASSIFIED ADVERTISING - Rooms for rent, furnished apartments..."],
                    max_length=None))
# ['Advertisement']

Pass max_length=None so the input text is not truncated.

Run both classifiers, not one. Neither visual nor textual signal alone classifies newspaper crops reliably.

How they connect

Field suffixes tell you a value's provenance:

Suffix	Meaning
_src	From the source archive or the scan filename
_ext	From an external source, such as a library catalog API
_gen	Generated by the IDI pipeline
_exp	Experimental or exploratory generation

The crop-type combination rule

crop_classification_gen is not simply the more confident of the two classifiers. The pipeline merges them like this, and you should apply the same rule when you classify a new crop yourself:

1. Default: take the prediction with the higher confidence score.
2. If the image classifier says Photograph or illustration with confidence above 0.5, take the image classifier.
3. If the image classifier says Empty with confidence above 0.4, take the image classifier.
4. If the text classifier says Empty, take the image classifier. Every crop with no OCR-able text is assigned Empty by rule with a confidence of exactly 1.0, so this value is a marker, not a measurement.
5. If only one modality produced a prediction, take it.

Both raw per-modality columns ship with the dataset, so you can apply a different rule instead.

Pipeline order for a new scan

scan -> segment into crops -> OCR each crop (Tesseract + dots.mocr) -> classify each crop (text + image) -> merge the two signals -> detect reading order -> enrich (language, text metrics, NER, subjects, thesauri, embeddings).

Gotchas

• crop_language_conf_gen is NULL when the language code was inherited from language_ext. The pipeline substitutes the issue-level code when detection confidence is below 0.50, when the crop holds fewer than 30 words, or when the issue language is not supported by Lingua. A missing confidence is the marker for an inherited code, not an error.
• Tesseract word boxes are relative to the crop; crop_bbox_gen is relative to the scan. Add the crop origin before drawing word boxes on a full scan.
• scan_image dominates row size. Never load it unless you need the pixels.
• The NER, subject, and thesauri fields are experimental. Subjects are a zero-shot soft ranking (mean top-1 confidence 0.65, and a mean gap of 0.457 to the second choice), so read crop_subject_gen as a ranking, not a label. Thesauri matches are naive keyword matches, useful as a navigational aid rather than as an interpretation of the text.
• Two subject labels overlap with crop types by design (Commercial Advertisements & Classifieds, Mastheads, Page Headers & Printer Imprints). They give the classifier somewhere to put crops that carry no topic.

Running the pipeline on a new collection

The pipeline runs 15 sequential steps over batches of issues and writes to SQLite. Every step is separate and can be re-run on its own.

git clone https://github.com/institutional/institutional-newspapers-pipeline.git
bash install.sh                                  # installs system-level dependencies too
nano .env                                         # corpora list and S3 credentials

uv run pipeline.py system cache-models            # pull models into the local cache
uv run pipeline.py system warmup-ocr-vlm          # pre-compile the OCR VLM encoder
uv run pipeline.py system build                   # list available issues
uv run pipeline.py orchestration prepare --corpus=CORPUS --items-per-batch=100
./run.sh <PIPELINE_RUN_ID>                        # execute, with logging
uv run pipeline.py orchestration status
uv run pipeline.py export CORPUS                  # build the releasable dataset

Every command has a --help option. Requirements: uv, Tesseract 5 with tessdata_best, sqlite, and a CUDA-capable GPU. vLLM only installs on Linux, so the VLM OCR step needs a Linux host. To add a corpus you need an S3 bucket of .tar.gz archives holding JP2, JPEG, or TIFF scans whose filenames sort alphabetically by page number; see "Adding corpora" in the pipeline README.

Auth & gating

export HF_TOKEN="hf_..."          # or: huggingface-cli login

Or load it from a local .env:

from dotenv import load_dotenv
load_dotenv()  # sets HF_TOKEN in the environment

Dependencies

pip install datasets huggingface_hub pyarrow pillow ultralytics model2vec python-dotenv