---
name: institutional-newspapers
description: "Use the Institutional Newspapers family of Hugging Face assets from the Institutional Data Initiative: historical newspaper scans segmented into individual crops, each enriched with OCR (Tesseract and VLM), crop-type classification, reading order, language detection, text metrics, named entities, subjects, and precomputed text and image embeddings. Currently covers the Boston Public Library collection (1.47M scans, 1795-1930), plus the three models that produced it and the production pipeline. Use when a task involves this corpus, reading or analyzing historical newspaper text or layout, reassembling an article from a page, finding advertisements or illustrations, tracking entities or subjects across newspapers, segmenting or classifying a new newspaper scan, or running the pipeline on another collection."
---

# Institutional Newspapers - dataset, models & pipeline

Institutional Newspapers is a growing family of Hugging Face assets from the [Institutional Data Initiative](https://institutional.org/): each instance takes one institution's historical newspaper collection, segments every scan into individual type-agnostic crops, and enriches each crop with OCR and analysis. New collections are added over time. Browse the [Institutional Newspapers collection](https://huggingface.co/collections/institutional/institutional-newspapers) to see what is available; a given instance may publish a different set of repos.

**This skill covers the instance published so far - the Boston Public Library collection:**

| Repo | Type | Contents |
| --- | --- | --- |
| `institutional/institutional-newspapers-bpl` | dataset | One row per scan. 1,473,635 scans, 83,147,041 crops |
| `institutional/institutional-newspapers-segmenter-yolo26x` | model | Scan -> crop bounding boxes |
| `institutional/institutional-newspapers-crop-classifier-image-yolo26m-cls` | model | Crop image -> crop type (7 classes) |
| `institutional/institutional-newspapers-crop-classifier-text-model2vec` | model | Crop text -> crop type (6 classes) |
| [institutional-newspapers-pipeline](https://github.com/institutional/institutional-newspapers-pipeline) | code | The 15-step pipeline that produced the dataset |

Scale: 1,473,635 public domain scans published between 1795 and 1930, 83,147,041 crops, 16.3 billion `o200k_base` tokens of VLM OCR text and 14.7 billion from Tesseract.

**Two ideas that drive every decision:**

1. **The `crop_*` columns are parallel lists that share one index, already in reading order.** One row is one scan. Every `crop_*` column holds one entry per crop, and all of them are sorted by the reading order the pipeline detected for that page. `zip` them to reassemble a crop. Filtering preserves reading order, because it only drops entries from lists that are already sorted.
2. **The dataset is precomputed pipeline output. Read the field instead of re-running the model.** All three models were already run over the whole collection and their results stored as columns. Only run a model on a **new** scan or crop that is not in the dataset.

## Pick the right asset

| I need to... | Use |
| --- | --- |
| Crops, OCR, and all enrichment for a scan in the collection | dataset `institutional-newspapers-bpl` (large - stream it) |
| Metadata only, without scan images, OCR text, or embeddings | Parquet column projection (see [below](#loading-only-specific-columns)) |
| The crop type of a crop (already computed) | field `crop_classification_gen` |
| The per-modality signal behind that decision | fields `crop_classification_image_only_gen` / `crop_classification_text_only_gen` (+ `_conf_gen`) |
| The reading order of a page (already computed) | the order of the `crop_*` lists themselves |
| Entities or subjects for a crop (already computed) | `crop_ner_per_gen` / `crop_ner_loc_gen` / `crop_ner_org_gen`, `crop_subject_gen` (experimental) |
| Vector search over crops | `crop_text_embeddings` (256-d) / `crop_image_embeddings` (384-d) |
| Find crops on a **new** newspaper scan | model `...-segmenter-yolo26x` |
| Classify a **new** crop | models `...-classifier-image-...` + `...-classifier-text-...`, then the [combination rule](#the-crop-type-combination-rule) |
| Process a **new** collection end to end | the [pipeline](#running-the-pipeline-on-a-new-collection) |

## Dataset

### `institutional-newspapers-bpl`

One row per newspaper scan (one page of one issue). A single `train` split. Large, because each row embeds its scan image - **always stream** unless you project columns. See [Auth & gating](#auth--gating).

```python
from datasets import load_dataset

ds = load_dataset(
    "institutional/institutional-newspapers-bpl",
    split="train",
    streaming=True,
    token=True,  # gated repo: uses HF_TOKEN / cached login
)

row = next(iter(ds))
print(row["issue_id_src"], row["page_number_gen"], row["year_ext"])
print(row["scan_image"])  # a PIL image, decoded from the embedded WEBP bytes

# The crop_* columns share one index and are already in reading order.
for bbox, category, text in zip(
    row["crop_bbox_gen"],
    row["crop_classification_gen"],
    row["crop_vlm_ocr_gen"],
):
    # One value per crop, NULL where the step produced no result.
    print(category or "", bbox, text or "")
```

Reassemble the readable text of a page by keeping only the crop types you want. The result stays in reading order:

```python
KEEP = {"Content", "Section heading"}

article = "\n\n".join(
    text
    for category, text in zip(row["crop_classification_gen"], row["crop_vlm_ocr_gen"])
    if category and category[0] in KEEP and text
)
```

## Loading only specific columns

Read metadata from the Parquet shards directly with column projection. Parquet is columnar, so the bytes of the skipped columns are never transferred. This is the difference between a small download and a very large one.

```python
import pyarrow.parquet as pq
from huggingface_hub import HfFileSystem

SKIP = {
    "scan_image",
    "crop_tesseract_ocr_gen",
    "crop_text_embeddings",
    "crop_image_embeddings",
}

fs = HfFileSystem(token=True)
shards = fs.glob("datasets/institutional/institutional-newspapers-bpl/**/*.parquet")

with fs.open(shards[0], cache_type="none") as handle:
    parquet_file = pq.ParquetFile(handle, pre_buffer=True)
    keep = [
        field.name
        for field in parquet_file.schema_arrow
        if field.name not in SKIP
    ]
    table = parquet_file.read(columns=keep)

print(table.num_rows, table.column_names)
```

Shards are named `BPL-part-NNNNN.parquet` and hold up to 250 scans each. They are compressed with zstd and written with a row-group size of 10.

## Models

All under `institutional/`. Run them in order on a **new** scan; for scans already in the collection, read the dataset instead.

### `...-newspapers-segmenter-yolo26x` (`ultralytics`)

A YOLO26x fine-tune. Single-class object detection: a scan in, one bounding box per crop out, regardless of crop type. Precision 0.927, recall 0.910, F1 0.918, mAP50 0.955, mAP50-95 0.901 on 153 held-out scans (2,991 bboxes).

```python
from huggingface_hub import snapshot_download
from ultralytics import YOLO

model_path = snapshot_download("institutional/institutional-newspapers-segmenter-yolo26x")
model = YOLO(f"{model_path}/best.pt")

results = model("newspaper_scan.jpg", imgsz=960, iou=0.15, conf=0.6)
boxes = [b.xyxy.tolist()[0] for b in results[0].boxes]  # crop these from the scan
```

Use `imgsz=960`, `iou=0.15`, `conf=0.6`. The confidence threshold drops low-certainty detections and the deliberately low IoU threshold suppresses heavily overlapping boxes, which is what these dense layouts need.

### `...-crop-classifier-image-yolo26m-cls` (`ultralytics`)

A YOLO26m-cls fine-tune. A **cropped** image in, one of the seven crop types out. Top-1 accuracy 0.913. Much stronger than the text classifier on the visual categories: F1 0.84 on `Photograph or illustration` against 0.48, and 0.92 on `Cartoon` against 0.68.

```python
model_path = snapshot_download("institutional/institutional-newspapers-crop-classifier-image-yolo26m-cls")
model = YOLO(f"{model_path}/best.pt")

r = model("crop.jpg", imgsz=768)[0]
print(r.names[r.probs.top1], float(r.probs.top1conf))
```

Use `imgsz=768`.

### `...-crop-classifier-text-model2vec` (`model2vec`)

A Model2Vec fine-tune of `minishlab/potion-base-32M` with a classifier head - light and CPU-friendly. The OCR text of a crop in, one of **six** crop types out. It does not learn `Empty`. Accuracy 0.92.

```python
from model2vec.inference import StaticModelPipeline

model = StaticModelPipeline.from_pretrained(
    "institutional/institutional-newspapers-crop-classifier-text-model2vec"
)
print(model.predict(["CLASSIFIED ADVERTISING - Rooms for rent, furnished apartments..."],
                    max_length=None))
# ['Advertisement']
```

Pass `max_length=None` so the input text is not truncated.

**Run both classifiers, not one.** Neither visual nor textual signal alone classifies newspaper crops reliably.

## How they connect

**Field suffixes** tell you a value's provenance:

| Suffix | Meaning |
| --- | --- |
| `_src` | From the source archive or the scan filename |
| `_ext` | From an external source, such as a library catalog API |
| `_gen` | Generated by the IDI pipeline |
| `_exp` | Experimental or exploratory generation |

### The crop-type combination rule

`crop_classification_gen` is not simply the more confident of the two classifiers. The pipeline merges them like this, and you should apply the same rule when you classify a new crop yourself:

1. Default: take the prediction with the higher confidence score.
2. If the image classifier says `Photograph or illustration` with confidence above **0.5**, take the image classifier.
3. If the image classifier says `Empty` with confidence above **0.4**, take the image classifier.
4. If the text classifier says `Empty`, take the image classifier. Every crop with no OCR-able text is assigned `Empty` by rule with a confidence of exactly 1.0, so this value is a marker, not a measurement.
5. If only one modality produced a prediction, take it.

Both raw per-modality columns ship with the dataset, so you can apply a different rule instead.

### Pipeline order for a new scan

scan -> segment into crops -> OCR each crop (Tesseract + dots.mocr) -> classify each crop (text + image) -> merge the two signals -> detect reading order -> enrich (language, text metrics, NER, subjects, thesauri, embeddings).

## Gotchas

- **`crop_language_conf_gen` is `NULL` when the language code was inherited from `language_ext`.** The pipeline substitutes the issue-level code when detection confidence is below 0.50, when the crop holds fewer than 30 words, or when the issue language is not supported by Lingua. A missing confidence is the marker for an inherited code, not an error.
- **Tesseract word boxes are relative to the crop; `crop_bbox_gen` is relative to the scan.** Add the crop origin before drawing word boxes on a full scan.
- **`scan_image` dominates row size.** Never load it unless you need the pixels.
- **The NER, subject, and thesauri fields are experimental.** Subjects are a zero-shot soft ranking (mean top-1 confidence 0.65, and a mean gap of 0.457 to the second choice), so read `crop_subject_gen` as a ranking, not a label. Thesauri matches are naive keyword matches, useful as a navigational aid rather than as an interpretation of the text.
- **Two subject labels overlap with crop types by design** (`Commercial Advertisements & Classifieds`, `Mastheads, Page Headers & Printer Imprints`). They give the classifier somewhere to put crops that carry no topic.

## Running the pipeline on a new collection

The pipeline runs 15 sequential steps over batches of issues and writes to SQLite. Every step is separate and can be re-run on its own.

```bash
git clone https://github.com/institutional/institutional-newspapers-pipeline.git
bash install.sh                                  # installs system-level dependencies too
nano .env                                         # corpora list and S3 credentials

uv run pipeline.py system cache-models            # pull models into the local cache
uv run pipeline.py system warmup-ocr-vlm          # pre-compile the OCR VLM encoder
uv run pipeline.py system build                   # list available issues
uv run pipeline.py orchestration prepare --corpus=CORPUS --items-per-batch=100
./run.sh <PIPELINE_RUN_ID>                        # execute, with logging
uv run pipeline.py orchestration status
uv run pipeline.py export CORPUS                  # build the releasable dataset
```

Every command has a `--help` option. Requirements: [uv](https://docs.astral.sh/uv/), Tesseract 5 with [`tessdata_best`](https://github.com/tesseract-ocr/tessdata_best), sqlite, and a CUDA-capable GPU. vLLM only installs on Linux, so the VLM OCR step needs a Linux host. To add a corpus you need an S3 bucket of `.tar.gz` archives holding JP2, JPEG, or TIFF scans whose filenames sort alphabetically by page number; see "Adding corpora" in the pipeline README.

## Auth & gating

```bash
export HF_TOKEN="hf_..."          # or: huggingface-cli login
```

Or load it from a local `.env`:

```python
from dotenv import load_dotenv
load_dotenv()  # sets HF_TOKEN in the environment
```

## Dependencies

```bash
pip install datasets huggingface_hub pyarrow pillow ultralytics model2vec python-dotenv
```