QraXAi modeli eklendi

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 coderian
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,163 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - en
+datasets:
+  - roneneldan/TinyStories
+tags:
+  - qraxai
+  - gpt2-tokenizer
+  - causal-lm
+  - tiny-stories
+  - from-scratch
+license: mit
+---
+
+# QraXAi-Basic-45M
+
+A small, decoder-only Transformer (~44.75M parameters) trained **from scratch** on
+[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) with the GPT-2 BPE
+tokenizer (50,257 tokens). QraXAi is a hand-written PyTorch GPT implementation
+(`model.py` / `configuration_qraxai.py`, shipped with this repo), not a fine-tune of GPT-2 —
+only the tokenizer is shared with GPT-2.
+
+This is an experimental research model, intended for learning/demo purposes.
+
+## Model details
+
+| | |
+|---|---|
+| Architecture | Decoder-only Transformer (GPT-style), pre-norm |
+| Parameters | **44,751,872** (~44.75M), all trainable |
+| Layers | 24 |
+| Hidden size | 256 |
+| Attention heads | 8 (head dim 32) |
+| Feed-forward | 4× hidden, GELU |
+| Context length | 256 tokens (hard limit) |
+| Vocabulary | 50,257 (GPT-2 BPE) |
+| Position encoding | Learned absolute embeddings |
+| Normalization | LayerNorm |
+| Weight tying | No (`lm_head` is separate) |
+| KV cache | No — generation recomputes the full context at every step |
+| Weights | fp32, 179 MB (`model.safetensors`) |
+| Special tokens | `bos = eos = <\|endoftext\|>` (id 50256) |
+| Custom code | Yes — requires `trust_remote_code=True` |
+
+Parameter breakdown: token embeddings 12.87M + position embeddings 0.07M +
+24 × 0.79M transformer blocks (18.95M) + final norm + untied `lm_head` 12.87M.
+
+## Quick start
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo_id = "coderian/QraXAi-Basic-45M"
+
+tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    repo_id,
+    trust_remote_code=True,
+    dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
+).to("cuda" if torch.cuda.is_available() else "cpu").eval()
+
+prompt = "Once upon a time, there was a little girl named Lily"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=244,          # prompt + new tokens must stay <= 256
+        do_sample=True,
+        temperature=0.8,
+        top_k=50,
+        top_p=0.95,
+        repetition_penalty=1.1,
+        pad_token_id=tokenizer.eos_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+Notes for generation:
+- **Context is a hard limit of 256 tokens.** The custom `forward` raises `ValueError`
+  if the sequence gets longer, so keep `len(prompt) + max_new_tokens <= 256`.
+- The model has **no KV cache**: each new token re-runs the full context, so generation
+  cost grows quickly with sequence length.
+- The custom `forward` does not use an attention mask for padding. Generate one prompt
+  at a time instead of batching.
+- Generation usually stops at `<|endoftext|>`, but because the training data is a
+  continuous stream of stories, the model sometimes starts a new story instead.
+
+## Example output
+
+Prompt: `Once upon a time, there was a little girl named Lily`
+(`temperature=0.8, top_k=50, top_p=0.95, repetition_penalty=1.1, max_new_tokens=244`):
+
+> Once upon a time, there was a little girl named Lily who loved to play in the big, green field. One day, she found a shiny stone on top of her backyard. She picked it up and showed it to her mom.
+>
+> "Look mommy, I found a pretty mineral!" said Lily excitedly. "It's very pretty!"
+>
+> Her mom smiled and said, "That's right, sweetie. It'll make sure you touch it. But remember, be careful with it because you might find something else inside."
+>
+> Lily nodded her head and kept playing with the jewel until she noticed that the box had fallen into a hole. She felt sad for her mom, but then remembered what her mom said about when something is hurt.
+>
+> The next day, Lily went back to the park and saw that the unknown stone was broken. She asked her mom if they could try and fix it. Her mom told her that it's okay to ask for help and that sometimes you can't use it without asking permission. So, Lily listened to her mom and never touched the stone again.
+> Once upon a time, there was a boy named Timmy. He loved to play with his toy car. One day, he went to
+
+(The last line is a new story the model began, cut off by the token limit.)
+
+## Training
+
+| | |
+|---|---|
+| Dataset | `roneneldan/TinyStories` (train split, streaming), first 130,000 stories |
+| Data size | 115.7M characters → 28.76M tokens → ~112,350 training blocks |
+| Objective | Next-token prediction (causal LM), cross-entropy |
+| Epochs | 1 (~7,000 optimizer steps) |
+| Batch size | 16 |
+| Block size | 256 |
+| Optimizer | AdamW, lr 3e-4 |
+| Gradient clipping | 1.0 |
+| Mixed precision | bf16 (on CUDA) |
+| Seed | 42 |
+
+## Files in this repository
+
+| File | Description |
+|---|---|
+| `config.json` | Model config (`GPTConfig` + `auto_map` for remote code) |
+| `model.safetensors` | fp32 weights (179 MB) |
+| `model.py` | `QraXAiForCausalLM` — custom modeling code |
+| `configuration_qraxai.py` | `GPTConfig` — custom configuration |
+| `tokenizer.json`, `tokenizer_config.json` | GPT-2 BPE tokenizer |
+| `generation_config.json` | Default generation settings |
+| `LICENSE` | MIT License |
+
+## Limitations
+
+- English only; trained exclusively on synthetic children's stories (TinyStories), so it
+  knows little about the real world.
+- Trained for a single epoch: grammar is mostly coherent, but content can be repetitive,
+  inconsistent or nonsensical.
+- Hard 256-token context; no sliding window, so long inputs must be truncated.
+- Not instruction-tuned — it does not follow instructions and is not a chat model.
+- No safety filtering or alignment of any kind. Do not use in production or for
+  user-facing applications.
+- TinyStories is synthetic data; nothing prevents the model from producing odd or
+  inappropriate continuations.
+
+## Acknowledgements
+
+- Dataset and idea: **TinyStories: How Small Can Language Models Be and Still Speak
+  Coherent English?** — Ronen Eldan and Yuanzhi Li, 2023 ([arXiv:2305.07759](https://arxiv.org/abs/2305.07759)).
+- Tokenizer: GPT-2 BPE (`gpt2`).
+- Built with PyTorch and Hugging Face Transformers.
+
+## License
+
+Released under the [MIT License](LICENSE). Note that the training dataset
+([TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)) has its own terms;
+check them if you plan to redistribute the data or use the model commercially.

config.json

@@ -0,0 +1,20 @@
+{
+  "architectures": [
+    "QraXAiForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_qraxai.GPTConfig",
+    "AutoModelForCausalLM": "model.QraXAiForCausalLM"
+  },
+  "bos_token_id": 50256,
+  "dtype": "float32",
+  "embed_dim": 256,
+  "eos_token_id": 50256,
+  "max_seq_len": 256,
+  "model_type": "qrax_ai",
+  "n_layers": 24,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.17.0",
+  "use_cache": false,
+  "vocab_size": 50257
+}

configuration_qraxai.py

@@ -0,0 +1,22 @@
+from transformers import PretrainedConfig
+
+class GPTConfig(PretrainedConfig):
+
+    model_type = "qrax_ai"
+
+    def __init__(
+        self,
+        vocab_size=10000,
+        n_layers=6,
+        max_seq_len=512,
+        embed_dim=256,
+        use_cache=False,
+        **kwargs
+    ):
+        super().__init__(**kwargs)
+
+        self.use_cache = use_cache
+        self.vocab_size = vocab_size
+        self.n_layers = n_layers
+        self.max_seq_len = max_seq_len
+        self.embed_dim = embed_dim

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.17.0",
+  "use_cache": false
+}

model.py

@@ -0,0 +1,243 @@
+from transformers import GenerationMixin, PreTrainedModel
+from transformers.modeling_outputs import CausalLMOutput
+import torch.nn as nn
+import torch
+import math
+
+try:
+    from .configuration_qraxai import GPTConfig
+except ImportError:
+    from configuration_qraxai import GPTConfig
+
+class CausalSelfAttention(nn.Module):
+
+    def __init__(self, embed_dim, num_heads):
+        super().__init__()
+
+        assert embed_dim % num_heads == 0, \
+            "embed_dim must be divisible by num_heads"
+
+        self.embed_dim = embed_dim
+        self.num_heads = num_heads
+        self.head_dim = embed_dim // num_heads
+
+        # Q, K, V projections
+        self.q_proj = nn.Linear(embed_dim, embed_dim)
+        self.k_proj = nn.Linear(embed_dim, embed_dim)
+        self.v_proj = nn.Linear(embed_dim, embed_dim)
+
+        self.o_proj = nn.Linear(embed_dim, embed_dim)
+
+    def forward(self, x):
+        batch_size, seq_len, embed_dim = x.shape
+
+        Q = self.q_proj(x)
+        V = self.v_proj(x)
+        K = self.k_proj(x)
+
+        Q = Q.view(
+            batch_size,
+            seq_len,
+            self.num_heads,
+            self.head_dim
+        )
+
+        K = K.view(
+            batch_size,
+            seq_len,
+            self.num_heads,
+            self.head_dim
+        )
+
+        V = V.view(
+            batch_size,
+            seq_len,
+            self.num_heads,
+            self.head_dim
+        )
+
+        Q = Q.transpose(1, 2)
+        K = K.transpose(1, 2)
+        V = V.transpose(1, 2)
+
+
+        scores = Q @ K.transpose(-2, -1)
+
+        scores = scores / math.sqrt(self.head_dim)
+
+        mask = torch.triu(
+            torch.ones(
+                seq_len,
+                seq_len,
+                device=x.device
+            ),
+            diagonal=1
+        ).bool()
+
+        scores = scores.masked_fill(
+            mask,
+            torch.finfo(scores.dtype).min
+        )
+
+        attention_w = torch.softmax(
+            scores,
+            dim=-1
+        )
+
+        output = attention_w @ V
+
+        output = output.transpose(1, 2)
+
+        output = output.contiguous().view(
+            batch_size,
+            seq_len,
+            embed_dim
+        )
+
+        # Final projection
+        output = self.o_proj(output)
+
+        return output
+
+
+class TransformerBlock(nn.Module):
+
+    def __init__(
+        self,
+        embed_dim
+    ):
+        super().__init__()
+
+        self.ln1 = nn.LayerNorm(embed_dim)
+
+        self.attention = CausalSelfAttention(embed_dim, 8)
+
+        self.ln2 = nn.LayerNorm(embed_dim)
+
+        # feed forward network
+        self.ffn = nn.Sequential(
+            nn.Linear(
+                in_features=embed_dim,
+                out_features=4*embed_dim
+            ),
+
+            nn.GELU(),
+
+            nn.Linear(
+                in_features=4*embed_dim,
+                out_features=embed_dim
+            )
+        )
+
+    def forward(self, x):
+        x = x + self.attention(
+            self.ln1(x)
+        )
+
+        x = x + self.ffn(
+            self.ln2(x)
+        )
+
+        return x
+
+class QraXAiForCausalLM(PreTrainedModel, GenerationMixin):
+
+    config_class = GPTConfig
+
+    def __init__(
+        self,
+        config
+    ):
+
+        super().__init__(config)
+
+        self.token_embedding = nn.Embedding(
+            config.vocab_size,
+            config.embed_dim
+        )
+
+        self.position_embedding = nn.Embedding(
+            config.max_seq_len,
+            config.embed_dim
+        )
+
+        self.transformer_blocks = nn.ModuleList([
+            TransformerBlock(config.embed_dim)
+            for _ in range(config.n_layers)
+        ])
+
+        self.ln_f = nn.LayerNorm(
+            config.embed_dim
+        )
+
+
+        self.lm_head = nn.Linear(
+            config.embed_dim,
+            config.vocab_size,
+            bias=False
+        )
+
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids,
+        labels=None,
+        **kwargs
+    ):
+
+        batch_size, seq_len = input_ids.shape
+
+        if seq_len > self.config.max_seq_len:
+            raise ValueError(
+                f"Sequence length ({seq_len}) "
+                f"cannot be greater than "
+                f"max_seq_len ({self.config.max_seq_len})"
+            )
+
+        positions = torch.arange(
+            seq_len,
+            device=input_ids.device
+        )
+
+        token_emb = self.token_embedding(
+            input_ids
+        )
+
+        pos_emb = self.position_embedding(
+            positions
+        )
+
+        x = token_emb + pos_emb
+
+        for block in self.transformer_blocks:
+            x = block(x)
+
+        x = self.ln_f(x)
+
+        logits = self.lm_head(x)
+
+        loss = None
+
+        if labels is not None:
+
+            shift_logits = logits[
+                :, :-1, :
+            ].contiguous()
+
+            shift_labels = labels[
+                :, 1:
+            ].contiguous()
+
+            loss = nn.functional.cross_entropy(
+                shift_logits.view(
+                    -1,
+                    shift_logits.size(-1)
+                ),
+                shift_labels.view(-1)
+            )
+
+        return CausalLMOutput(
+            loss=loss,
+            logits=logits
+        )

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4bfeaa3c57234fef408a5c51c9463df89eb67ca3e3c8c7e7517d2d6d4bbd6de
+size 179049920

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1024,
+  "pad_token": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}