Upload folder using huggingface_hub

README.md

@@ -0,0 +1,160 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+- en
+tags:
+- causal-lm
+- biinduct
+- pretraining
+- matched-compute
+- the-pile
+- 125m
+- balanced
+---
+
+# Bi-Induct 125M Balanced
+
+This repository contains the **Bi-Induct 125M Balanced** checkpoint from *Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learning*.
+
+This release corresponds to the **0.13B** setting in the paper and is a **research checkpoint** intended for studying matched-compute pretraining, induction-style curricula, and in-context learning behavior. It is **not** instruction-tuned, alignment-tuned, or safety-tuned.
+
+## Variant
+
+Bi-Induct balanced curriculum. Each synthetic injection chooses forward-copy or backward-copy with equal probability.
+
+## Model overview
+
+- Architecture: decoder-only Transformer
+- Positional encoding: RoPE (`theta=10000`)
+- Normalization: pre-norm residual blocks
+- MLP: SwiGLU
+- Attention: grouped-query / grouped key-value attention
+- Precision: bfloat16 training
+- Context length: 1024
+- Embeddings: untied input/output embeddings
+
+## Model specification
+
+| Field | Value |
+|---|---:|
+| Parameters (paper label) | 0.13B |
+| Layers | 12 |
+| Hidden size | 768 |
+| Intermediate / MLP size | 3,072 |
+| Head dimension | 64 |
+| Attention heads | 12 |
+| KV heads | 3 |
+
+## Training data
+
+All checkpoints in this family were pretrained on the **deduplicated THE PILE** in streaming / shuffled mode. A stable MD5-based hash was used to create a fixed held-out evaluation slice, with **0.2% of the corpus** reserved for evaluation (roughly **0.4B tokens**). Tokenization was truncated to **1024 tokens per sequence**.
+
+For the Bi-Induct variants, synthetic snippets were interleaved on top of the natural stream:
+
+- **Induction**: `[S || SEP || S]`
+- **Anti-Induction**: `[S || SEP || reverse(S)]`
+- **Balanced**: each injection randomly chooses induction or anti-induction
+
+The main cross-scale experiments used **span length L = 20** and **initial mix ratio m0 = 50%**, linearly annealed to zero over the full training budget.
+
+## Training recipe
+
+- Optimizer: AdamW (`beta1=0.9`, `beta2=0.999`, weight decay `0.1`)
+- Learning rate: peak `1e-3`
+- Schedule: `3%` linear warmup, then cosine decay
+- Update size: `2^16` tokens per update
+- Token budget: approximately `20N` tokens following the Chinchilla-style rule of thumb
+- Comparison protocol: iso-FLOPs across curricula at each scale
+
+## Evaluation summary for the 125M family
+
+The table below summarizes the main results at this scale. Standard LM benchmarks are evaluated **3-shot** and Todd et al. function-style probes are evaluated **10-shot** with **HITS@1**.
+
+| Variant | Standard LM ICL composite ↑ | Todd-style ICL composite ↑ | Held-out PPL ↓ |
+|---|---:|---:|---:|
+| Baseline | 22.7 ± 0.5 | 5.3 ± 0.9 | 21.8 |
+| Induction | 21.9 ± 0.5 | 4.1 ± 0.7 | 25.8 |
+| Anti-Induction | 22.5 ± 0.4 | 3.8 ± 0.7 | 26.2 |
+| Balanced | 22.4 ± 0.6 | 5.2 ± 0.8 | 26.2 |
+
+**This checkpoint:** **Balanced**.
+
+## Benchmarks included
+
+### Standard LM benchmarks
+- MMLU
+- Winogrande
+- CommonSenseQA
+- PIQA
+- HellaSwag
+- TriviaQA-Wiki
+- BBH (CoT)
+- OpenBookQA
+- ARC-Challenge
+- GPQA
+- GSM-8K
+- MathQA
+- BoolQ
+- LAMBADA
+
+### Todd et al. function-style probes
+- alphabetically first 3
+- alphabetically first 5
+- alphabetically last 3
+- alphabetically last 5
+- capitalize
+- capitalize first letter
+- capitalize last letter
+- choose first of 3
+- choose first of 5
+- choose last of 3
+- choose last of 5
+- choose middle of 3
+- choose middle of 5
+- lowercase first letter
+- lowercase last letter
+- next capital letter
+- next item
+- prev item
+- word length
+
+## Example usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+repo_id = "MohammedSabry/biinduct-125m-balanced"
+
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = AutoModelForCausalLM.from_pretrained(repo_id)
+
+prompt = "The capital of France is"
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Limitations
+
+- These are research checkpoints, not production chat models.
+- They were designed to study the relationship between induction-style telemetry and load-bearing ICL behavior under matched compute.
+- The synthetic interventions are intentionally lightweight and token-level; results should not be interpreted as ruling out richer data-rewrite strategies.
+- Because Bi-Induct replaces a fraction of natural data under iso-FLOPs, some trade-offs may reflect natural-text displacement in addition to mechanistic redundancy.
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{sabry2026inductionsignaturesenoughmatchedcompute,
+      title={Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learning}, 
+      author={Mohammed Sabry and Anya Belz},
+      year={2026},
+      eprint={2509.22947},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2509.22947}, 
+}
+```
+

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "MistralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 3,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.52.4",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.52.4"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5dc6ffafdb9b3c5f73ac13a6e731fefe60e304a5569eaa4204f7a8e3401d00a7
+size 303613576

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}