zhongweixie/inplace-ttt-qwen3-4b-32k-ttt

Qwen3-4B with In-Place TTT (32k, 6000 steps)

This model is Qwen3-4B trained with In-Place Test-Time Training (In-Place TTT) on 32k context length for 6000 training steps.

🎯 Model Overview

• Base Model: Qwen3-4B (Qwen2.5-4B architecture)
• Training Method: In-Place Test-Time Training (TTT)
• Context Length: 32k tokens
• Training Steps: 6000
• Model Size: 8.4GB
• Format: HuggingFace Transformers (safetensors)
• Training Date: May 13, 2026

📊 What is In-Place TTT?

In-Place Test-Time Training is a novel approach that enables:

• Efficient long-context modeling without full attention
• Test-time adaptation to input sequences
• Memory-efficient training compared to standard long-context methods
• Better performance on long-context tasks

For more details, see the paper: In-Place Test-Time Training

🚀 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "zhongweixie/inplace-ttt-qwen3-4b-32k-ttt",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(
    "zhongweixie/inplace-ttt-qwen3-4b-32k-ttt",
    trust_remote_code=True
)

# Generate text
prompt = "Your long context prompt here..."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📈 Training Details

• Training Data: ProLong pretokenized dataset
• Context Length: 32,768 tokens
• Training Steps: 6,000
• Batch Size: Micro batch size 4, Global batch size 64
• Optimizer: AdamW
• Hardware: 8x H100 GPUs

🔗 Related Models

This is part of the In-Place TTT model family:

• Qwen3-4B 32k Baseline - Baseline without TTT
• Qwen3-4B 64k TTT - 64k context version
• Qwen3-4B 64k Baseline - 64k baseline

📦 Resources

• GitHub Repository: https://github.com/zhongweixie/inplace_ttt
• Results Dataset: https://huggingface.co/datasets/zhongweixie/inplace-ttt-results
• Paper: https://arxiv.org/abs/2604.06169

📄 License

Apache 2.0 License

🎓 Citation

@inproceedings{feng2026inplace,
  title     = {In-Place Test-Time Training},
  author    = {Feng, Guhao and Luo, Shengjie and Hua, Kai and Zhang, Ge and Huang, Wenhao and He, Di and Cai, Tianle},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  note      = {Oral Presentation},
  url       = {https://arxiv.org/abs/2604.06169}
}

📞 Contact

For questions or issues, please open an issue in the GitHub repository.

---

Model Card Author: Hansirui (zhongweixie)
Uploaded: 2026-09-04
Project: In-Place Test-Time Training