metadata
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
pipeline_tag: text-generation
library_name: mlx
language:
- nog
tags:
- mlx
- lora
- peft
- low-resource
- nogai
- turkic
- continued-pre-training
datasets:
- ansarzeinulla/Nogai-Unified-Corpus-v1
Qwen2.5-1.5B-Nogai-LoRA (Phase 1: continued pre-training)
An mlx-lm LoRA adapter for Qwen2.5-1.5B-Instruct, trained on raw Nogai text (Nogai-Unified-Corpus-v1, 9.8 M Qwen2.5 tokens). This is Phase 1 of NogaiLLM. It learns Nogai spelling and morphology, but after this phase the model no longer follows chat instructions: it continues text like a newspaper. For translation, use Phase 2 (Qwen2.5-1.5B-Nogai-SFT-Experimental) on top of this adapter.
Training (from adapter_config.json)
| Base | MLX copy of Qwen2.5-1.5B-Instruct |
| LoRA | rank 8, scale 20, dropout 0 |
| Trained layers | q/k/v/o/gate/up/down projections of layers 12–27 (16 blocks), 5.28 M parameters |
| Batch / iterations / learning rate | 2 / 2,500 / 2e-4, seed 0 |
| Max sequence length | 512 tokens |
| Hardware | Apple M2 Pro, 16 GB, mlx-lm |
Results
Reported in the first version of the paper, on the corpus validation split:
| Model | Per-token perplexity | TD (digraphs / 100 words) |
|---|---|---|
| Qwen2.5-1.5B-Instruct (base) | 191.13 | 17.16 |
| + this adapter | 12.14 | 15.05 |
| Human Nogai text (reference) | — | 22.7 |
Notes:
- TD counts the Nogai digraphs аь/оь/уь/нъ per 100 words. It is only meaningful next to the human value (22.7). By that measure, this adapter's output is further from natural Nogai than the base model's.
- Per-token perplexity can't be compared across tokenizers. These numbers are being re-measured as bits per byte with
eval_bits_per_byte.py.
Usage (Apple Silicon / MLX)
The model is a text continuer after this phase, so skip the chat template:
pip install mlx-lm
mlx_lm.generate --model Qwen/Qwen2.5-1.5B-Instruct \
--adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
--prompt "Буьгуьнги куьн" --max-tokens 128 --temp 0.5 --ignore-chat-template
To build the base for Phase 2, fuse this adapter into the model:
mlx_lm.fuse --model Qwen/Qwen2.5-1.5B-Instruct \
--adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
--save-path local_qwen_1.5B_Nogai_Base
Citation
@misc{zeinulla2026nogaillm,
title = {NogaiLLM: Parameter-Efficient Continued Pre-Training and Catastrophic Forgetting in Zero-Resource Turkic Languages},
author = {Zeinulla, Ansar},
year = {2026},
note = {Manuscript under revision. Code: https://github.com/ansarzeinulla/NogaiLLM-Apple-Silicon}
}