---
license: apache-2.0          # أو أي ترخيص تفضله (يمكنك تغييره)
language:
  - en
library_name: peft
tags:
  - causal-lm
  - text-generation
  - lora
  - gpt2
  - fine-tuned
base_model: openai-community/gpt2
pipeline_tag: text-generation
---

# GPT-2 FineWeb (machkour's continued pretraining variant)

## Model Description

This is a **continued pretraining** (fine-tuning for next-token prediction) of the original **GPT-2** (small, 124M parameters) on a high-quality subset of **FineWeb** dataset.

- **Base model**: openai-community/gpt2 (original GPT-2 small)
- **Training method**: PEFT + LoRA (r=32, alpha=32, targets: c_attn, c_proj, c_fc)
- **Quantization during training**: 4-bit NF4 + double quant + bfloat16 compute
- **Training objective**: Causal language modeling (next token prediction)
- **Purpose**: Improve general text continuation / next-word prediction quality before personality / instruction tuning in the next stage.

## When & How Was It Created?

- **Creation date**: February 26, 2026
- **Training duration**: ≈ 60 minutes (600 training steps)
- **Hardware**: Google Colab T4 GPU (15 GB VRAM)
- **Training start → end**: ~39–40 minutes wall-clock time for the final run (after map/tokenization overhead)
- **Created by**: @younes
## Training Data

- **Dataset**: HuggingFaceFW/fineweb — configuration: `sample-10BT`
- **Processed subset**: 700,000 documents (after filtering short texts)
- **Total tokens**: ≈ 416 million tokens (estimated from previous runs)
- **Format**: `.jsonl.gz` with single field `{"text": "..."}`
- **Language**: Primarily English (fineweb is English-dominant)
- **Data source date**: Mostly web crawl snapshots from ~2013 (old but very clean high-quality subset)

**Note**: This is **not** instruction-tuned or chat-tuned. It is still a raw language model optimized for free-form text continuation.

## Model Size & Files

- **Base parameters**: 124M (GPT-2 small)
- **Trainable LoRA parameters**: ≈ 4.72M (3.65% of total)
- **Total effective parameters**: still 124M (LoRA is additive)
- **Disk size (saved folder)**: ~250–350 MB (4-bit base + LoRA adapters)
- **Saved directory**: `gpt2-fineweb-final`
- **Main files**:
  - `adapter_config.json`
  - `adapter_model.bin` (or safetensors)
  - `config.json`
  - `generation_config.json`
  - `pytorch_model.bin` (quantized base)
  - `tokenizer.json`, `vocab.json`, `merges.txt`

## How to Use / Load the Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "openai-community/gpt2"
adapter_path = "machkour/gpt2-fineweb-416M-tokens"   # ← change to your repo after upload

tokenizer = AutoTokenizer.from_pretrained(base_model_name)

model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    torch_dtype="auto"
)

model = PeftModel.from_pretrained(model, adapter_path)

# Optional: merge LoRA weights into base model (for faster inference)
# model = model.merge_and_unload()

# Example generation
prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.85,
    top_p=0.92
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Intended Use & Limitations

- Best for: open-ended text generation, story continuation, code / prose completion
- Not suitable (yet) for: chat / instruction following, Q&A, Arabic-dominant tasks
- Next planned step: personality / role injection + instruction tuning

## Training Hyperparameters (final run)

- Optimizer: adamw_8bit
- Learning rate: 5e-5
- Batch size: 4 × 4 (effective 16)
- Steps: 600
- Warmup steps: 50
- Gradient checkpointing: yes
- Mixed precision: bf16

## Results / Observations

- Loss decreased from ~3.89 → ~3.54 over 600 steps
- Visible improvement in fluency and topical coherence compared to vanilla GPT-2
- Still shows signs of repetition / old-web style (due to FineWeb age)

## How to Cite / Reference

If you use this model:

```
@misc{gpt2-fineweb-machkour,
  author       = {machkour},
  title        = {GPT-2 continued on FineWeb (416M tokens)},
  year         = {2026},
  month        = {February},
  howpublished = {\url{https://huggingface.co/machkour/gpt2-fineweb-416M-tokens}},
}
```

---

Good luck with the upload!

After saving this as README.md in your model folder, you can push everything with:

```python
from huggingface_hub import login, upload_folder

login()  # paste your HF token

upload_folder(
    folder_path="gpt2-fineweb-final",
    repo_id="machkour/gpt2-fineweb-416M-tokens",
    repo_type="model",
    commit_message="Upload fine-tuned GPT-2 on FineWeb"
)
```

(Install `huggingface_hub` if needed: `!pip install huggingface_hub`)