metadata
library_name: transformers
license: other
base_model: dogtooth/open-lm-3b-202301
tags:
- llama-factory
- full
- sft
- chrononauts
- temporal-lm
model-index:
- name: open-lm-3b-202301-sft-nontemporal
results: []
Open LM 3B January 2023 — SFT (Non-Temporal)
This model is a supervised fine-tuned (SFT) version of dogtooth/open-lm-3b-202301, trained on the non-temporal split of the mattwang123/chrononauts-sft-filtered dataset.
It is part of the Chrononauts project (JHU CLSP), which studies temporal/time-continual language modeling using Apple Open LM models with different knowledge cutoff dates. Based on the TiC-LM paper.
Model Details
| Property | Value |
|---|---|
| Base model | dogtooth/open-lm-3b-202301 |
| Parameters | 2.7B |
| Architecture | LLaMA-style (pre-norm, SwiGLU, RoPE) |
| Knowledge cutoff | January 2023 |
| Context length | 2,048 tokens |
| Vocab size | 50,432 |
| Training data | mattwang123/chrononauts-sft-filtered (non_temporal split) |
| Eval data | HellaSwag |
| Train loss | 0.923 |
| Eval loss | 2.602 |
Chat Template
This model uses a simple Human: / Assistant: prompt format with <|endoftext|> as the separator token.
Format
Human: {user_message}<|endoftext|>
Assistant:{assistant_response}<|endoftext|>
For multi-turn conversations:
Human: {first_message}<|endoftext|>
Assistant:{first_response}<|endoftext|>
Human: {second_message}<|endoftext|>
Assistant:{second_response}<|endoftext|>
With a system prompt:
System: {system_message}<|endoftext|>
Human: {user_message}<|endoftext|>
Assistant:{assistant_response}<|endoftext|>
Usage with Transformers
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "dogtooth/open-lm-3b-202301-sft-nontemporal"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
# Using the chat template
messages = [
{"role": "user", "content": "What is the capital of France?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# Result: "Human: What is the capital of France?<|endoftext|>\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
Manual Prompting
prompt = "Human: What is the capital of France?<|endoftext|>\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
Training Hyperparameters
- Learning rate: 1e-5
- Batch size: 2 per device x 4 GPUs x 4 gradient accumulation = 32 effective
- Epochs: 3
- Scheduler: Cosine with 10% warmup
- Precision: bf16
- DeepSpeed: ZeRO Stage 2
- Hardware: 4x NVIDIA A100 80GB
Framework Versions
- Transformers 4.57.1
- PyTorch 2.6.0+cu124
- Datasets 4.0.0
- Tokenizers 0.22.2