# Model Card: TheoBERT Base

## Model Details

- **Model name:** TheoBERT Base (`theo-bert-base`)
- **Model type:** Bidirectional encoder with masked language modeling head
- **Architecture:** 12-layer Transformer with RoPE, ReLU² MLP, gated value embeddings, residual interpolation
- **Parameters:** 273,051,864 (≈273M)
- **Vocabulary:** 30,522 tokens (inherited from `google-bert/bert-base-uncased` tokenizer)
- **Context length:** 256 tokens (rotary cache supports up to 2,560)
- **License:** Apache 2.0
- **Contact:** [Toran Billups](https://huggingface.co/toranb)

## Uses

### Direct Use

TheoBERT Base is a masked language model for biblical and theological text. It is designed for:

- **Fill-mask inference** — predicting masked tokens in biblical-domain sentences
- **Token-level scoring** — computing log-probabilities of candidate tokens in context
- **Encoder hidden states** — extracting contextual embeddings for downstream task heads

### Downstream Use

This model is suitable as a base for fine-tuning on domain-specific tasks such as:

- Named entity recognition (biblical figures, places, events)
- Text classification (theological topic, genre, author attribution)
- Sequence labeling (part-of-speech tagging for biblical languages, when paired with appropriate tokenizers)
- Continued pretraining or domain adaptation

### Out-of-Scope Use

- **Text generation** — This is an encoder, not a decoder. It cannot autoregressively generate text.
- **General-domain NLP** — Trained exclusively on biblical and theological text. Performance on news, social media, or scientific text is untested and likely degraded.
- **Languages other than English** — Training data is English-language biblical material.

## Bias, Risks, and Limitations

### Domain Specialization

The model is trained on a narrow domain (biblical and theological English text). Its representations are shaped by the theological traditions present in the training corpus. Users should be aware that:

- The model's predictions reflect the distributional patterns of the training data, not ground-truth theological positions.
- Different theological traditions use the same terms with different meanings. The model may favor the usage patterns most prevalent in its training data.
- Tokenization is inherited from `bert-base-uncased`, which was trained on general English. Multi-piece subword splits on theological terms (e.g., "propitiation" → "prop", "##iti", "##ation") may degrade performance on rare vocabulary.

### Training Data

The training data is private and consists of:

- 270,000 sequences from bible text, Christian books, biblical commentaries, and synthetic data
- Corpus composition is not publicly auditable, which limits external bias analysis

### Known Failure Modes

- **Old Testament proper-noun recall** — Names like Jeremiah, Jonah, Job, and Nebuchadnezzar are weakly recalled
- **Multi-piece subword reconstruction** — Long words spanning many wordpieces (e.g., "sabachthani", "iniquity") have higher error rates
- **Canonical knowledge** — Lowest per-category performance (88.4%) relative to doctrinal categories

### Environmental Impact

Training was conducted over 49 epochs total (24 + 25) on a single GPU. Carbon footprint was not directly measured but is modest relative to large-scale LLM training. The released fp16 weights (546 MB) reduce download bandwidth and inference memory.

## Training Details

### Training Data

Private corpus of 270,000 sequences drawn from:

- Bible text (multiple English translations)
- Christian theological books and commentaries
- Synthetic data generated for domain coverage

No public dataset identifiers are available.

### Training Procedure

**Stage 1 — MLM Pretraining (24 epochs)**

- Objective: Masked language modeling
- Sequence length: 256 tokens
- Single-piece token masking
- Final train loss: 1.0679
- Final train accuracy: 76.42%

**Stage 2 — Continued Pretraining with Whole-Word Masking (25 epochs)**

- Objective: Whole-word masked language modeling at 18% mask rate
- Initialized from Stage 1 checkpoint
- Final train loss: 0.8958
- Final train accuracy: 79.66%

### Optimizer

Custom dual-optimizer setup combining AdamW (for embeddings, MLM head, residual lambdas, value gate parameters) and Muon (momentum-orthogonalized Newton-Schulz optimizer) for linear layer weight matrices.

### Hyperparameters

- AdamW betas: (0.8, 0.95)
- Muon momentum: 0.95
- Embedding learning rate: 0.3
- Matrix learning rate: 0.02
- Weight decay: 0.0
- Precision: fp32 training, fp16 release

## Evaluation

Evaluated on a 546-case domain-specific MLM benchmark. See [`EVAL.md`](EVAL.md) for full methodology, test case examples, and per-category breakdown.

| Metric | Value |
|---|---|
| Overall pass rate | 94.7% (517 / 546) |
| Difficulty-weighted | 94.6% |
| Easy | 94.9% |
| Medium | 94.9% |
| Hard | 94.2% |

## Technical Specifications

| Property | Value |
|---|---|
| Framework | PyTorch 2.4+ / Transformers 5.2.0 |
| Format | safetensors (fp16) |
| Model size (disk) | 546 MB |
| Model size (fp32 memory) | ~1.0 GB |
| Tokenizer | `google-bert/bert-base-uncased` (bundled, see Attribution) |
| Custom code required | Yes (`trust_remote_code=True`) |

### Architecture Detail

- 12 transformer blocks, hidden size 768, 8 attention heads (head dim 96)
- RoPE (Rotary Position Embedding) applied to Q and K projections
- RMS normalization on Q and K (no learnable gain)
- ReLU-squared MLP activation (`relu(x)²`)
- Gated value embeddings on even-indexed layers (0, 2, 4, 6, 8, 10)
- Learned residual interpolation: `λ_resid · block_output + λ_x0 · initial_embedding`
- MLM head: `Linear → GELU → RMSNorm → Linear`

## Attribution

This repository redistributes the fast-tokenizer files (`tokenizer.json`, `tokenizer_config.json`) from [`google-bert/bert-base-uncased`](https://huggingface.co/google-bert/bert-base-uncased) unmodified. Those files are released by Google under the Apache License 2.0 and are bundled here so `AutoTokenizer.from_pretrained` and the Hugging Face Hub `fill-mask` widget work without an extra download step. TheoBERT Base is itself released under Apache 2.0; see `LICENSE`.

## Citation

```bibtex
@misc{theobertbase2026,
  author       = {Toran Billups},
  title        = {TheoBERT Base: A Domain-Specialized Masked Language Model for Biblical and Theological Text},
  year         = 2026,
  publisher    = {Hugging Face},
  howpublished = {https://huggingface.co/toranb/theo-bert-base}
}
```