---
language:
- tr
license: apache-2.0
tags:
- llama
- causal-lm
- slm
- micro-model
- experimental
- turkish
- tr-llm
- AhiskaAI
- AhiskaTurks
- MeskhetianTurks
- AhıskaTürkleri
datasets:
- BILGEM-AI/BILGE-Synthetic-Math
- BILGEM-AI/BILGE-Synthetic-Web
- BILGEM-AI/BILGE-Synthetic-Stories
pipeline_tag: text-generation
---

# AhıskaAI-10M-Experimental-v0.1

**AhıskaAI-10M-Experimental-v0.1** is an ultra-compact ~10 million parameter Small Language Model (SLM) trained from scratch. As part of the AhıskaAI experimental series, this micro-model is designed to test the absolute lower limits of Turkish language representations, syntax learning, and synthetic task understanding at minimal computational costs.

## Experimental Scope & Setup

- **Objective:** Evaluate how much linguistic structure and reasoning a 10M parameter micro-model can capture when trained on dense synthetic data.
- **Training Epochs:** Pre-trained for **1 full epoch** across a ~3.5 GB synthetic Turkish corpus.
- **Vocabulary Efficiency:** Uses an optimized 8,000 token vocabulary specifically suited for micro-scale models.

## Model Details

- **Architecture:** `LlamaForCausalLM` (Micro MHA)
- **Parameters:** ~10M
- **Hidden Size:** 288
- **Intermediate Size:** 768
- **Attention Heads:** 8 (Query) / 8 (Key/Value)
- **Layers:** 8
- **Context Length:** 512 tokens
- **Vocabulary Size:** 8,000
- **Precision:** `float32`

## Pre-training Dataset Mixture

The model was pre-trained on a synthetic Turkish corpus (~3.5 GB total):

| Data Source | Approx. Size | Primary Focus | Link |
| :--- | :--- | :--- | :--- |
| **BILGEM AI Synthetic Web** | ~2.0 GB | General Web, Knowledge & Grammar | [BILGEM-AI/BILGE-Synthetic-Web](https://huggingface.co/datasets/BILGEM-AI/BILGE-Synthetic-Web) |
| **BILGEM AI Synthetic Math** | ~750 MB | Mathematical Reasoning & Logic | [BILGEM-AI/BILGE-Synthetic-Math](https://huggingface.co/datasets/BILGEM-AI/BILGE-Synthetic-Math) |
| **BILGEM AI Synthetic Stories** | ~750 MB | Narrative Flow & Text Generation | [BILGEM-AI/BILGE-Synthetic-Stories](https://huggingface.co/datasets/BILGEM-AI/BILGE-Synthetic-Stories) |

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AhıskaAI/AhıskaAI-10M-Experimental-v0.1-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

prompt = "Türkiye"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### About AhıskaAI
**AhıskaAI** is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.