---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: xlr8harder/talkie-1930-13b-base-tf
datasets:
- xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m
tags:
- transformers
- safetensors
- bfloat16
- custom_code
- text-generation
- talkie
- yarn
- long-context
- pre-1931
- alternate-checkpoint
---

# Talkie 1930 13B YaRN 32k From 4k Step1000

This is an alternate 32k-context YaRN extension of
[`xlr8harder/talkie-1930-13b-base-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-base-tf).
It uses an 8x YaRN extension from a 4,096-token starting context and continued
pretraining to step1000 on the same Project Gutenberg pre-1931 data recipe used
for the main Talkie YaRN 32k checkpoint.

The recommended checkpoint from this experiment series is
[`xlr8harder/talkie-1930-13b-yarn-32k-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-tf).
That model used a 16x extension from the 2,048-token reference config and the
step500 checkpoint. It performed better overall at 16k and 32k, and avoided the
severe variable-tracking collapse seen in this 4k-start run.

## License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See
[`LICENSE`](./LICENSE). The continued-pretraining corpus has separate dataset
provenance and licensing documented at
[`xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`](https://huggingface.co/datasets/xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m).

## Checkpoint Family

| Checkpoint | Role |
| --- | --- |
| [`talkie-1930-13b-yarn-32k-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-tf) | Recommended 2k-start step500 checkpoint |
| [`talkie-1930-13b-yarn-32k-step1000-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-step1000-tf) | Final 2k-start checkpoint |
| [`talkie-1930-13b-yarn-32k-from4k-step500-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-from4k-step500-tf) | 4k-start step500 comparison checkpoint |
| [`talkie-1930-13b-yarn-32k-from4k-step1000-tf`](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-from4k-step1000-tf) | This checkpoint |

## Usage

This model uses custom Talkie modeling/tokenization code, so load it with
`trust_remote_code=True`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-from4k-step1000-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
```

For vLLM, set `--max-model-len 32768` and enable remote code.

## RULER Comparison

Scores are aggregate RULER accuracy percentages from our harness, using 100
examples per task and greedy decoding. It is unclear how much RULER
unintentionally penalizes Talkie because Talkie is intentionally limited to
pre-1931 training data while some RULER tasks involve modern entities and facts;
the effect is hard to quantify here, but it is likely non-zero.

| Model / setup | 2k | 4k | 8k | 16k | 32k |
| --- | ---: | ---: | ---: | ---: | ---: |
| Talkie base, extrapolation only | 85.86 | 77.71 | 23.40 | n/a | n/a |
| Talkie YaRN 32k, 2k start, step500 | 80.78 | 79.50 | 73.15 | 70.05 | 61.83 |
| Talkie YaRN 32k, 4k start, step500 | 83.80 | 80.71 | 75.64 | 68.80 | 54.76 |
| Talkie YaRN 32k, 4k start, step1000 | 84.18 | 80.98 | 76.17 | 68.45 | 55.01 |

This checkpoint is included for comparison because it tests the later-discovered
4k starting context. It is better at short-context RULER tiers than the 2k-start
checkpoint, but worse at the longest tiers. At 32k it scored 55.01 overall, with
`vt` at 0.40, compared with 61.83 overall and `vt` at 26.20 for the recommended
2k-start step500 checkpoint.

## Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; `qa_2` exceeded the 2k context budget
in this RULER setup and was excluded by the length constraint for that tier.

| Task | 2k | 4k | 8k | 16k | 32k |
| --- | ---: | ---: | ---: | ---: | ---: |
| Overall | 84.18 | 80.98 | 76.17 | 68.45 | 55.01 |
| `cwe` | 27.30 | 33.40 | 20.10 | 16.40 | 8.90 |
| `fwe` | 36.67 | 49.33 | 45.67 | 43.33 | 19.33 |
| `niah_multikey_1` | 100.00 | 100.00 | 99.00 | 95.00 | 92.00 |
| `niah_multikey_2` | 100.00 | 100.00 | 100.00 | 99.00 | 94.00 |
| `niah_multikey_3` | 92.00 | 84.00 | 79.00 | 39.00 | 7.00 |
| `niah_multiquery` | 99.25 | 99.00 | 96.75 | 91.25 | 88.50 |
| `niah_multivalue` | 98.00 | 93.75 | 81.75 | 85.50 | 47.00 |
| `niah_single_1` | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| `niah_single_2` | 100.00 | 100.00 | 100.00 | 97.00 | 100.00 |
| `niah_single_3` | 99.00 | 98.00 | 96.00 | 74.00 | 56.00 |
| `qa_1` | 70.00 | 76.00 | 66.00 | 65.00 | 52.00 |
| `qa_2` | n/a | 48.00 | 52.00 | 52.00 | 50.00 |
| `vt` | 88.00 | 71.20 | 54.00 | 32.40 | 0.40 |

## Training Recipe

The training data was
[`xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`](https://huggingface.co/datasets/xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m),
a Project Gutenberg split filtered to English public-domain books with
publication years 1500-1930, totaling 265,080,702 Talkie tokens. Training used
BF16 FSDP on one 8xA100 80GB node, 8 FSDP ranks, 32,768-token sequences,
262,144 tokens per step, 1000 max steps, cosine LR decay from `1e-5` to `1e-6`,
50 warmup steps, and weight decay `0.01`.