---
language:
  - en
license: apache-2.0
tags:
  - gpt
  - text-generation
  - causal-lm
  - pytorch
  - safetensors
  - custom-trained
---

# gpt-model-2-decoder-100000-tiny-stories-fp16

A custom GPT-style language model trained from scratch using PyTorch.

## Model Details

| Parameter | Value |
|-----------|-------|
| Architecture | GPT (Decoder-only Transformer) |
| Hidden size (`d_model`) | 768 |
| Attention heads | 8 |
| Transformer blocks | 1 |
| Max sequence length | 1024 |
| Vocabulary size | 32000 |
| Dropout | 0.2 |

## Tokenizer

Custom BPE tokenizer trained with the HuggingFace `tokenizers` library.

**Special tokens:** `<|endoftext|>` · `<|pad|>` · `<|unk|>`

## Quick Start

You can easily load this model and tokenizer using the `transformers` library. Because the model uses a custom architecture, you must pass `trust_remote_code=True`.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("sdkjfgndjfg/gpt-model-2-decoder-100000-tiny-stories-fp16", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("sdkjfgndjfg/gpt-model-2-decoder-100000-tiny-stories-fp16", trust_remote_code=True)

# Set up device
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

# Generate text
prompt = "The transformer is based on"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
output_ids = model.generate(
    **inputs, 
    max_new_tokens=50, 
    do_sample=True, 
    temperature=0.8,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

## Training Details

- **Optimizer**: AdamW (lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
- **Scheduler**: CosineAnnealingLR (eta_min=1e-5)
- **Loss**: CrossEntropyLoss (next-token prediction)
- **Gradient clipping**: max_norm=1.0

## License

Apache 2.0