mehmettozlu/Turkish-Qwen-2.5-7B-Instruct-GGUF

🇹🇷 Turkish Qwen-2.5-7B-Instruct (GGUF)

This repository contains the GGUF formatted version of the Qwen2.5-7B-Instruct model, which has been fine-tuned on a Turkish instruction dataset using Unsloth.

The model is optimized to act as a highly capable Turkish AI assistant running locally and 100% offline via llama.cpp.

💾 Available GGUF File and System Requirements

Currently, the most optimal and highly requested quantization version is available in this repository:

File Name	Size	Recommended RAM	Description
*q4_k_m.gguf	~4.3 GB	8 GB	🔥 The Golden Standard. Offers the best balance between inference speed and model intelligence. Ideal for running on standard consumer hardware.

---

📊 Model Performance Benchmarks (LLM-as-a-Judge)

This model has been tested under identical conditions alongside other popular Turkish GGUF models and evaluated via an LLM-as-a-Judge benchmark to measure Turkish language proficiency, instruction-following capabilities, and coding performance.

🏆 Comparison Table

Model	Parameters	Geography (10)	Email Formatting (10)	Coding (10)	Total	Performance Summary
Qwen-2.5-Instruct	7B	4.0	8.5	10.0	22.5 / 30	Most Balanced: Flawless Python code, fluent Turkish, and high instruction adherence. Minor hallucination tendencies on local geographical data.
Llama-3.1-Instruct	8B	0.5	0.0	5.0	5.5 / 30	Partial Success: Strong algorithmic background (generates working code), but suffers from severe token repetition and looping on text tasks.
Mistral-NeMo-Instruct	12B	2.0	2.0	1.0	5.0 / 30	Weak Instruction Following: While grammar is readable, it lacks task orientation (generates a list instead of code, fails to formalize casual tone).
Gemma-2-IT	9B	0.0	0.0	0.0	0.0 / 30	Format Incompatibility: Due to special token structures and quantization sensitivity, it fails to produce meaningful output and enters a repetition loop.

💬 Prompt Template (ChatML)

The Qwen 2.5 architecture uses the ChatML format. To prevent hallucinations and get the best performance, you must use the following structure when interacting with the model via API or code:

<|im_start|>system
Sen yardımsever bir Türkçe asistansın.<|im_end|>
<|im_start|>user
[Write your prompt here]<|im_end|>
<|im_start|>assistant

🚀 How to Run the Model

You can run this model locally with complete privacy and zero internet connection required.

Option 1: Using Python (llama-cpp-python)

If you want to integrate the model into your own Python applications or RAG pipelines, you can easily use the llama-cpp-python library.

1. Install the library via pip:

pip install llama-cpp-python

(Note: If you want GPU acceleration, refer to the llama-cpp-python documentation for cuDNN/Metal installation commands).

2. Download model from Hugging Face

wget -O Turkish-Qwen-2.5-7B-Instruct-q4_k_m.gguf https://huggingface.co/mehmettozlu/Turkish-Qwen-2.5-7B-Instruct-GGUF/resolve/main/Qwen2.5-7B-Instruct.Q4_K_M.gguf

3. Create a Python script (run.py) and use the following code:

from llama_cpp import Llama

# 1. Load the model
llm = Llama(
    model_path="./Turkish-Qwen-2.5-7B-Instruct-q4_k_m.gguf", # Path to your downloaded GGUF file
    n_ctx=2048,          # Context window size
    n_gpu_layers=-1      # Offload all layers to GPU (set to 0 if using only CPU)
)

# 2. Chat with the model
response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "Sen yardımsever bir Türkçe asistansın."},
        {"role": "user", "content": "Yapay zeka modellerinin nasıl eğitildiğini kısaca açıklar mısın?"}
    ]
)

# 3. Print the assistant's response
print(response["choices"][0]["message"]["content"])

Option 2: Using LM Studio (Easiest - GUI)

• Download and install LM Studio.
• In the application's search bar, type mehmettozlu/Turkish-Qwen-2.5-7B-Instruct-GGUF.
• Download the q4_k_m.gguf file from the search results.
• Navigate to the "Chat" tab on the left panel, load the model from the top dropdown, and start chatting in Turkish.

Option 3: Using Ollama (For Developers)

If you have Ollama installed, you can call the model directly from your terminal or integrate it into your local projects (e.g., LangChain, RAG pipelines) by creating a custom Modelfile.

Create a text file named Modelfile (no extension) on your computer and paste the following content:

# Note: Replace the filename below if yours is slightly different in the repo
FROM hf.co/mehmettozlu/Turkish-Qwen-2.5-7B-Instruct-GGUF/Turkish-Qwen-2.5-7B-Instruct-q4_k_m.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """Sen yardımsever bir Türkçe asistansın."""
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

• Open your terminal and build the model:

ollama create turkish-qwen -f Modelfile

• Run the model:

ollama run turkish-qwen

Option 4: Using llama.cpp (For CLI / Terminal Enthusiasts)

If you prefer the raw performance and lightweight nature of the terminal, you can run the model directly using llama.cpp:

1. Clone the llama.cpp repository to your machine and build it (run make on Linux/macOS, or follow CMake instructions for Windows).
2. Download the *q4_k_m.gguf file from this repository and place it in your llama.cpp folder.
3. Open your terminal and start an interactive chat session with the following command:

./llama-cli -m Turkish-Qwen-2.5-7B-Instruct-q4_k_m.gguf -c 2048 --conversation --color

(Note: If you are using an older version of llama.cpp, the executable might be named ./main instead of ./llama-cli. The --conversation flag automatically detects the ChatML prompt format embedded in the GGUF file.)