institutional-books.md

---
name: institutional-books
description: "Use the Institutional Books family of Hugging Face datasets and models: the public-domain book corpus and its metadata, enriched/segmented text, and per-page visual elements, plus models for visual-element detection, classification, orientation, and topic classification. Use when a task involves the Institutional Books corpus, digitized-book text or metadata analysis, extracting or classifying figures/charts/music/illustrations from scanned pages, or assigning library topics to books."
---

# Institutional Books — datasets & models

A family of Hugging Face assets built from *Institutional Books* (983K public-domain books, ~242B tokens, digitized from Harvard Library's Google Books participation and refined by the Institutional Data Initiative). 
Technical reports: arXiv:2506.08300, arXiv:2608.19026, arXiv:2608.18957 

**Two ideas that drive every decision:**

1. **`barcode_src` is the join key.** One book = one barcode. Every dataset carries `barcode_src`, so you relate them with a join, not a re-download.
2. **The datasets are precomputed model outputs.** The topic and visual-element models were already run over the whole corpus and their results stored as columns. **Read the field instead of re-running the model.** Only run a model on *new* images or text that isn't already in the corpus.

## Pick the right asset

| I need to… | Use |
| --- | --- |
| Book text (original + post-processed OCR) + all metadata | dataset `institutional-books-hl` (large — stream it) |
| Metadata only (title, author, dates, language, topic, OCR scores…) without downloading text | dataset `institutional-books-hl-metadata` |
| Structurally segmented text (front/middle/back matter) + refined text stats | dataset `institutional-books-hl-enriched-text` |
| Figures/charts/music/illustrations already extracted from pages (boxes, labels, orientation, captions, embeddings) | dataset `institutional-books-hl-visual-elements` |
| A book's high-level topic (already computed) | field `topic_or_subject_gen` in the corpus/metadata datasets |
| A book's visual elements (already computed) | dataset `institutional-books-hl-visual-elements` |
| Topic of a **new** book from its metadata | model `institutional-books-topic-classifier-bert` |
| Find visual elements on a **new** page scan | model `…-visual-elements-detection` → `…-classification` → `…-orientation` |

## Datasets

All under the `institutional/` org. Load with the `datasets` library. See [Auth & gating](#auth--gating) for gated repos.

### `institutional-books-hl` — full corpus
One row per book (~983K). Key fields: `barcode_src` (primary key), `title_src`, `author_src`, `date1_src`, `language_src`/`language_gen`, `topic_or_subject_gen`, `text_by_page_src` (original OCR, `List[str]` by page), `text_by_page_gen` (post-processed OCR; ~850K books in eng/deu/fra/ita/spa). Large — **always stream**.

```python
from datasets import load_dataset

ds = load_dataset("institutional/institutional-books-hl", split="train",
                  streaming=True, token=True)  # token=True → uses HF_TOKEN / cached login
for row in ds:
    print(row["barcode_src"], row["title_src"])
    print(row["text_by_page_gen"][0])  # first page, post-processed OCR
    break
```

### `institutional-books-hl-metadata` — metadata only
Same schema as the corpus **minus** the `text_by_page_*` text columns. Use this whenever you don't need the full text — it's far smaller and ungated.

```python
ds = load_dataset("institutional/institutional-books-hl-metadata", split="train")
```

### `institutional-books-hl-enriched-text` — segmented text
One row per book. `barcode_src` + `frontmatter_gen` / `middlematter_gen` / `backmatter_gen` (structural segments), `processed_middlematter_gen`, plus text-quality stats (`tokenizability_ratio_gen`, bits-per-byte `bpb_*`, n-gram/word/sentence counts). Use when you need document structure or refined-text quality signals.

### `institutional-books-hl-visual-elements` — extracted visual elements
One row per **detected element** (10M–100M rows) — the detection→classification→orientation pipeline already run over the corpus. Key fields:

- `id`, `barcode_src`, `page_filename_src`, `bbox_xyxy_gen` — where the element is
- `crop_gen` — the cropped image
- `detection_confidence_gen`
- `classification_gen` (+ `_confidence_gen`, `_probs_gen`) — matches the split it came from
- `orientation_correction_gen` (+ `_confidence_gen`, `_probs_gen`)
- `caption_exp`, `phash_gen`, `embedding_gen` — caption, perceptual hash, vector

**Splits are the categories**, so select one instead of filtering: `music`, `image_illustration`, `chart_graph`, `ex_libris_decorative`, `artifact`, `other`.

```python
from pathlib import Path

out = Path("music_crops"); out.mkdir(exist_ok=True)
ve = load_dataset("institutional/institutional-books-hl-visual-elements",
                  split="music", streaming=True)  # pick the category you want
for row in ve.take(20):                            # first 20 music scores
    row["crop_gen"].save(out / f"{row['id']}.png")  # unique filename per element (PIL image)
    print(row["id"], row["barcode_src"], row["detection_confidence_gen"])
```

## Models

All under `institutional/`. Weights via `huggingface_hub.hf_hub_download`.

### `institutional-books-topic-classifier-bert` (`transformers`)
mBERT fine-tune. Input: a formatted metadata string. Output: one of 20 Library of Congress top-level classes (SCIENCE, LAW, MEDICINE, HISTORY OF THE AMERICAS, …). This is the source of `topic_or_subject_gen` — only run it for books not in the corpus.

```python
from transformers import pipeline

pipe = pipeline("text-classification",
                model="institutional/institutional-books-topic-classifier-bert")
text = "Title: A treatise on analytical geometry\nAuthor: Hymers, J.\nYear: 1848\nLanguage: English"
print(pipe(text))  # [{'label': 'SCIENCE', 'score': 0.99...}]  (all fields optional)
```

### Visual-elements pipeline
Three models mirror the visual-elements dataset columns. Run them in order on a **new** page scan; for corpus pages, read the dataset instead.

**1. Detection** — `…-visual-elements-detection` (YOLOv26n, `ultralytics`). Page scan → bounding boxes (single class "visual element").

```python
from huggingface_hub import hf_hub_download
from ultralytics import YOLO

det = YOLO(hf_hub_download("institutional/institutional-books-visual-elements-detection",
                           "weights/best.pt"))
results = det.predict("page_scan.jpg", imgsz=640, conf=0.3, iou=0.2)
boxes = [b.xyxy.tolist()[0] for b in results[0].boxes]  # crop these from the page
```

**2. Classification** — `…-visual-elements-classification` (YOLOv26s-cls, `ultralytics`). A **cropped** element → one of 5 classes: `0 Artifact · 1 Chart/Graph · 2 Ex Libris/Decorative · 3 Image/Illustration · 4 Music`.

```python
cls = YOLO(hf_hub_download("institutional/institutional-books-visual-elements-classification",
                           "weights/best.pt"))
r = cls.predict("crop.jpg", imgsz=640)[0]
print(r.names[r.probs.top1], float(r.probs.top1conf))
```

**3. Orientation** — `…-visual-elements-orientation` (EfficientNetV2-M, raw PyTorch `state_dict`). A crop → rotation correction: `0 upright · 1 rotate_90_clockwise · 2 rotate_180 · 3 rotate_90_counterclockwise`. Predictions below **0.99** confidence default to `upright`.

```python
import torch, torch.nn as nn, torchvision.models as models
from torchvision import transforms
from PIL import Image
from huggingface_hub import hf_hub_download

m = models.efficientnet_v2_m(weights=None)
m.classifier = nn.Sequential(nn.Dropout(0.3, inplace=True),
                             nn.Linear(m.classifier[1].in_features, 4))
m.load_state_dict(torch.load(
    hf_hub_download("institutional/institutional-books-visual-elements-orientation",
                    "weights/weights.pth"), map_location="cpu", weights_only=True))
m.eval()

prep = transforms.Compose([
    transforms.Resize((512, 512)), transforms.CenterCrop(480), transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
labels = {0: "upright", 1: "rotate_90_clockwise", 2: "rotate_180", 3: "rotate_90_counterclockwise"}

with torch.no_grad():
    probs = torch.softmax(m(prep(Image.open("crop.jpg").convert("RGB")).unsqueeze(0)), 1)[0]
top = int(probs.argmax())
label = labels[top] if float(probs[top]) >= 0.99 else "upright"

# apply: rotate_90_clockwise → img.rotate(-90, expand=True);
#        rotate_180 → 180; rotate_90_counterclockwise → 90
```

## How they connect

**Field suffixes** tell you a value's provenance:

| Suffix | Meaning |
| --- | --- |
| `_src` | From the source collection (original bibliographic/OCR data) |
| `_gen` | Generated by IDI analysis or a model (e.g. `topic_or_subject_gen`) |
| `_ext` | Pulled from an external source (e.g. `hathitrust_data_ext`) |
| `_exp` | Experimental (e.g. `caption_exp`) |

**Join on `barcode_src`** to combine datasets — e.g. filter visual elements by a book's language, date, or topic:

```python
meta = load_dataset("institutional/institutional-books-hl-metadata", split="train")
fr = {r["barcode_src"] for r in meta if r["language_gen"] == "fra"}  # French books

ve = load_dataset("institutional/institutional-books-hl-visual-elements",
                  split="train", streaming=True)
music_in_french = (r for r in ve
                   if r["barcode_src"] in fr and r["classification_gen"] == "Music")
```

Within the visual-elements dataset, `page_filename_src` + `bbox_xyxy_gen` locate an element on its specific page.

**Visual pipeline** (only for new scans): page → detection (boxes) → crop → classification (5 classes) + orientation (4 classes) → optional caption/embedding. This is exactly what produced the visual-elements dataset columns.

## Auth & gating

Set a token that has been granted access, then pass it to the loaders:

```bash
export HF_TOKEN="hf_..."          # or: huggingface-cli login
```

## Dependencies

```bash
pip install datasets huggingface_hub transformers torch torchvision pillow ultralytics
```