ansarzeinulla
/

Qwen2.5-1.5B-Nogai-LoRA

@@ -12,130 +12,66 @@ tags:
 - low-resource
 - nogai
 - turkic
-- kazakh
-- cross-lingual
-- continuous-pre-training
 datasets:
 - ansarzeinulla/Nogai-Unified-Corpus-v1
-metrics:
-- perplexity
 ---
-<div align="center">
-# ⚡️ Qwen2.5-1.5B-Nogai-LoRA (Phase 1 Baseline)
-### Experimental Continuous Pre-Training (CPT) Adapter for Zero-Resource Turkic NLP
-[![Paper](https://img.shields.io/badge/Paper-ACM%20TALLIP-B31B1B.svg)](#)
-[![Framework](https://img.shields.io/badge/Framework-Apple%20MLX-000000.svg?logo=apple&logoColor=white)](https://github.com/ml-explore/mlx)
-[![Base Model](https://img.shields.io/badge/Base-Qwen2.5--1.5B--Instruct-blue.svg)](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct)
-[![Dataset](https://img.shields.io/badge/Dataset-Nogai--Unified-green.svg)](https://huggingface.co/datasets/ansarzeinulla/Nogai-Unified-Corpus-v1)
-*A parameter-efficient low-rank adaptation establishing the mathematical baseline for Nogai morphological mapping.*
-</div>
-<br>
-## 🔬 Model Overview
-This is a **Quantized LoRA (QLoRA) adapter** for `Qwen2.5-1.5B-Instruct`, engineered entirely on constrained consumer silicon via the Apple `mlx-lm` framework.
-This model represents **Phase 1** of the NogaiLLM adaptation pipeline. It was trained exclusively on the unstructured text of the `Nogai-Unified-Corpus-v1` (~4 million tokens) for 2,500 iterations. Its primary objective was to evaluate the resilience of dense Byte-Pair Encoding (BPE) architectures to zero-resource Cyrillic vocabulary expansion.
----
-## ⚠️ Scientific Disclaimer: Catastrophic Forgetting
-**This adapter is released strictly for academic reproducibility and baseline measurement.** It is *not* a conversational agent.
-Because this adapter was trained via Continuous Pre-Training (CPT) on raw, unstructured journalistic text without an accompanying instruction-tuning phase, it exhibits severe **Catastrophic Forgetting**.
-*   **The Success:** The multi-head attention layers successfully mapped the local Turkic morphology, dropping perplexity from a raw 191.13 to an optimized **12.14**. The model correctly generates Nogai-specific Cyrillic digraphs (e.g., `нъ`, `аь`, `оь`, `уь`).
-*   **The Failure State:** The unstructured training completely overwrote the base model's latent instruction-following manifold. When presented with standard conversational templates (e.g., `ChatML`), the model's predictive routing collapses. It acts solely as an unconditional text generator, aggressively mimicking newspaper formatting rather than answering prompts.
-*(Note: To test the fully recovered, instruction-following model, please refer to the Phase 2 SFT model: [Nogai-SFT-Experimental](https://huggingface.co/ansarzeinulla/Qwen2.5-1.5B-Nogai-SFT-Experimental)).*
----
-## 📊 Empirical Benchmarks & Mode Collapse
-This adapter was evaluated against Western-centric models in a rigorous 6-model architectural ablation study. The results prove that rigid, Latin-optimized tokenizers (Llama) cannot be force-mapped onto zero-resource Turkic languages via LoRA alone, whereas Eastern architectures (Qwen) achieve successful structural transfer.
-| Model Configuration | Scale | Test Loss ($\downarrow$) | Perplexity ($\downarrow$) | Nogai Typographic Density ($\uparrow$) |
-| :--- | :---: | :---: | :---: | :---: |
-| Qwen-2.5-0.5B (Raw Base) | 0.5B | 5.401 | 221.62 | 0.72% |
-| **Qwen-2.5-0.5B (Adapted)** | 0.5B | 3.379 | 29.33 | **22.95%** |
-| Qwen-2.5-1.5B (Raw Base) | 1.5B | 5.253 | 191.13 | 17.16% |
-| **Qwen-2.5-1.5B (Adapted)** | 1.5B | **2.497** | **12.14** | **15.05%*** |
-| Llama-3.2-1B (Raw Base) | 1.0B | 5.836 | 342.44 | 89.66%** |
-| Llama-3.2-1B (Adapted) | 1.0B | 5.505 | 245.82 | 0.00% *(Mode Collapse)* |
-*\*Represents morphologically accurate digraph placement, free of hallucination loops.*
-*\*\*Artifact: Llama base exhibited severe OOD fragmentation, falling into an autoregressive Cyrillic repetition loop.*
----
-## 🚀 Usage (Apple Silicon / MLX)
-Because this model acts as an unconditional morphological generator rather than a chat model, **you must bypass the chat template** during generation.
-### CLI Execution
-Install the MLX framework natively on macOS:
 ```bash
 pip install mlx-lm
-```
-Run generation with the `--ignore-chat-template` flag to prevent format-induced hallucination:
-```bash
-mlx_lm.generate \
-    --model Qwen/Qwen2.5-1.5B-Instruct \
     --adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
-    --prompt "Буьгуьнги куьн" \
-    --max-tokens 128 \
-    --temp 0.5 \
-    --ignore-chat-template
 ```
-### Python API Integration
-```python
-from mlx_lm import load, generate
-# Load the base model and dynamically apply the Phase 1 LoRA weights
-model, tokenizer = load(
-    "Qwen/Qwen2.5-1.5B-Instruct",
-    adapter_path="ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA"
-)
-# Unconditional text completion (Do NOT apply chat templates)
-prompt = "Ногай районынынъ орталыгы Терекли-Мектеб авлында"
-response = generate(
-    model,
-    tokenizer,
-    prompt=prompt,
-    max_tokens=100,
-    verbose=True
-)
-print(response)
 ```
----
-## 📚 Citation
-If you analyze this adapter's weights, utilize the ablation data, or study our mode collapse findings, please cite the core ACM TALLIP / arXiv paper:
 ```bibtex
-@article{zeinulla2026nogaillm,
-  title={NogaiLLM: Parameter-Efficient Continuous Pre-Training and Architectures of Catastrophic Forgetting in Zero-Resource Turkic Languages},
-  author={Zeinulla, Ansar},
-  journal={arXiv preprint arXiv:2607.xxxxx},
-  year={2026},
-  publisher={Nazarbayev University / ACM TALLIP}
 }
 ```
-<div align="center">
-  <i>Maintained by <a href="https://huggingface.co/ansarzeinulla">@ansarzeinulla</a> • Nazarbayev University</i>
-</div>

 - low-resource
 - nogai
 - turkic
+- continued-pre-training
 datasets:
 - ansarzeinulla/Nogai-Unified-Corpus-v1
 ---
+# Qwen2.5-1.5B-Nogai-LoRA (Phase 1: continued pre-training)
+An `mlx-lm` LoRA adapter for Qwen2.5-1.5B-Instruct, trained on raw Nogai text ([Nogai-Unified-Corpus-v1](https://huggingface.co/datasets/ansarzeinulla/Nogai-Unified-Corpus-v1), 9.8 M Qwen2.5 tokens). This is Phase 1 of [NogaiLLM](https://github.com/ansarzeinulla/NogaiLLM-Apple-Silicon). It learns Nogai spelling and morphology, but after this phase the model **no longer follows chat instructions**: it continues text like a newspaper. For translation, use Phase 2 ([Qwen2.5-1.5B-Nogai-SFT-Experimental](https://huggingface.co/ansarzeinulla/Qwen2.5-1.5B-Nogai-SFT-Experimental)) on top of this adapter.
+## Training (from `adapter_config.json`)
+| | |
+|---|---|
+| Base | MLX copy of Qwen2.5-1.5B-Instruct |
+| LoRA | rank 8, scale 20, dropout 0 |
+| Trained layers | q/k/v/o/gate/up/down projections of layers 12–27 (16 blocks), 5.28 M parameters |
+| Batch / iterations / learning rate | 2 / 2,500 / 2e-4, seed 0 |
+| Max sequence length | 512 tokens |
+| Hardware | Apple M2 Pro, 16 GB, `mlx-lm` |
+## Results
+Reported in the first version of the paper, on the corpus validation split:
+| Model | Per-token perplexity | TD (digraphs / 100 words) |
+|---|---|---|
+| Qwen2.5-1.5B-Instruct (base) | 191.13 | 17.16 |
+| + this adapter | 12.14 | 15.05 |
+| Human Nogai text (reference) | — | 22.7 |
+Notes:
+- **TD** counts the Nogai digraphs аь/оь/уь/нъ per 100 words. It is only meaningful next to the human value (22.7). By that measure, this adapter's output is *further* from natural Nogai than the base model's.
+- **Per-token perplexity can't be compared across tokenizers.** These numbers are being re-measured as bits per byte with [`eval_bits_per_byte.py`](https://github.com/ansarzeinulla/NogaiLLM-Apple-Silicon/blob/main/evaluation/eval_bits_per_byte.py).
+## Usage (Apple Silicon / MLX)
+The model is a text continuer after this phase, so skip the chat template:
 ```bash
 pip install mlx-lm
+mlx_lm.generate --model Qwen/Qwen2.5-1.5B-Instruct \
     --adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
+    --prompt "Буьгуьнги куьн" --max-tokens 128 --temp 0.5 --ignore-chat-template
 ```
+To build the base for Phase 2, fuse this adapter into the model:
+```bash
+mlx_lm.fuse --model Qwen/Qwen2.5-1.5B-Instruct \
+    --adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
+    --save-path local_qwen_1.5B_Nogai_Base
 ```
+## Citation
 ```bibtex
+@misc{zeinulla2026nogaillm,
+  title  = {NogaiLLM: Parameter-Efficient Continued Pre-Training and Catastrophic Forgetting in Zero-Resource Turkic Languages},
+  author = {Zeinulla, Ansar},
+  year   = {2026},
+  note   = {Manuscript under revision. Code: https://github.com/ansarzeinulla/NogaiLLM-Apple-Silicon}
 }
 ```