README.md

---
license: apache-2.0
language:
- en
- zh
library_name: transformers
pipeline_tag: text-generation
base_model: BlinkDL/rwkv7-g1
tags:
- rwkv
- rwkv7
- goose
- linear-attention
- recurrent
---

# RWKV-7 "Goose" G1h (reasoning line) 7.2B — ready for the in-tree `transformers` implementation

This is **not a new model**. It is a format conversion of
[`BlinkDL/rwkv7-g1`](https://huggingface.co/BlinkDL/rwkv7-g1) →
`rwkv7-g1h-7.2b-20260710-ctx10240.pth`, laid out as a `transformers` directory so the `rwkv7`
implementation can load it with `from_pretrained`. All credit for the weights
belongs to **BlinkDL / the RWKV project**; they are redistributed here under the
Apache-2.0 licence they were released under.

The conversion is a rename, not a transformation: every one of the 1062 tensors
in the source `.pth` is **bit-identical** here (verified tensor by tensor). No tensors were
added: this checkpoint natively carries layer 0's value-residual LoRA at its
initialization values (layer 0 never reads it -- it *produces* `v_first`), and it
is preserved bit-identically rather than dropped.
`bfloat16`, the dtype the source is stored in. Training context length 10240.

## Usage

The implementation ships inside this repo (`auto_map` remote code), so stock
`transformers` is all you need — no fork, no extra package:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("RWKV/RWKV7-Goose-World2.8-0.1B-HF", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Hakureirm/rwkv7-g1h-7.2b-hf", trust_remote_code=True, dtype=torch.bfloat16)

inputs = tokenizer("The Eiffel Tower is located in the city of", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=20)[0]))
```

(The tokenizer is the RWKV "world" tokenizer, shared by every model in the family,
so the 0.1B repo above is just a convenient place to load it from.)

The same model class is also open as an in-tree PR
([rwkv-rs/transformers-rwkv#2](https://github.com/rwkv-rs/transformers-rwkv/pull/2));
this repo is the way to use it today on released transformers.

## Runs on

Pure-PyTorch portable path, no CUDA-only code on it. The implementation is verified
greedy 20/20 against BlinkDL's own runtime on **CPU**, **Apple Silicon MPS** and
**CUDA** (see [`Hakureirm/rwkv7-0.1b-hf`](https://huggingface.co/Hakureirm/rwkv7-0.1b-hf)
for that matrix and the logit-level comparison). Optional Triton kernels engage only
on CUDA and fall back to the portable path everywhere else.

RWKV-7 is attention-free and fully recurrent: the state is a fixed-size matrix per
head, so there is no KV cache, memory is constant in context length, and each new
token costs the same as the first.

## What was verified for this size

- all 1062 source tensors **bit-identical** after conversion, none missing, none extra;
- the converter's shape check: every tensor matches the shape the inferred config
  implies (32 layers, hidden 4096, 64 heads x 64, ffn 16384, LoRA ranks w/a/v/g 128/128/96/480);
- a bf16 CUDA greedy smoke run. `"The Eiffel Tower is located in the city of"` continues:

  > The Eiffel Tower is located in the city of Paris, France. It is situated on the Champ de Mars, a large public

The logit-level check against BlinkDL's own runtime was run on the 0.1B conversion
(same converter, same implementation), not repeated per size.

## Reproducing the conversion

```bash
huggingface-cli download BlinkDL/rwkv7-g1 rwkv7-g1h-7.2b-20260710-ctx10240.pth --local-dir .
python src/transformers/models/rwkv7/convert_rwkv7_checkpoint_to_hf.py \
    --checkpoint rwkv7-g1h-7.2b-20260710-ctx10240.pth \
    --flavour native --dtype bfloat16 --output_dir ./rwkv7-g1h-7.2b-hf
```

The converter compares the checkpoint's tensor shapes against the ones the config
implies and refuses a disagreement, so a config that names a different model fails
rather than producing something that loads and generates noise.

## Citation

The model is RWKV-7 "Goose" by Bo Peng (BlinkDL) and the RWKV community. Reference
implementation: [BlinkDL/RWKV-LM](https://github.com/BlinkDL/RWKV-LM).