mehmettozlu/Turkish-Llama-3.1-8B-Instruct-GGUF

🇹🇷 Turkish Llama-3.1-8B-Instruct (GGUF)

This repository contains the GGUF formatted version of the Meta-Llama-3.1-8B-Instruct model, which has been fine-tuned on a Turkish instruction dataset using Unsloth.

The model is optimized to act as a highly capable Turkish AI assistant running locally and 100% offline via llama.cpp, Ollama, and LM Studio.

💾 Available GGUF Files and System Requirements

We provide multiple quantization options. The q4_k_m version is highly recommended for consumer hardware:

File Name	Size	Recommended RAM	Description
*q4_k_m.gguf	~4.9 GB	8 GB	🔥 The Golden Standard. Offers the best balance between inference speed and model intelligence. Ideal for running on standard consumer hardware.
*q5_k_m.gguf	~5.7 GB	10 GB	Higher quality with a slight trade-off in generation speed.
*q8_0.gguf	~8.5 GB	12 GB	Near-lossless original quality. Requires more RAM/VRAM.

---

📊 Model Performance Benchmarks (LLM-as-a-Judge)

This model has been tested under identical conditions alongside other popular Turkish GGUF models and evaluated via an LLM-as-a-Judge benchmark to measure Turkish language proficiency, instruction-following capabilities, and coding performance.

🏆 Comparison Table

Model	Parameters	Geography (10)	Email Formatting (10)	Coding (10)	Total	Performance Summary
Qwen-2.5-Instruct	7B	4.0	8.5	10.0	22.5 / 30	Most Balanced: Flawless Python code, fluent Turkish, and high instruction adherence. Minor hallucination tendencies on local geographical data.
Llama-3.1-Instruct	8B	0.5	0.0	5.0	5.5 / 30	Partial Success: Strong algorithmic background (generates working code), but suffers from severe token repetition and looping on text tasks.
Mistral-NeMo-Instruct	12B	2.0	2.0	1.0	5.0 / 30	Weak Instruction Following: While grammar is readable, it lacks task orientation (generates a list instead of code, fails to formalize casual tone).
Gemma-2-IT	9B	0.0	0.0	0.0	0.0 / 30	Format Incompatibility: Due to special token structures and quantization sensitivity, it fails to produce meaningful output and enters a repetition loop.

💬 Prompt Template (Llama 3.1 Format)

Llama 3.1 uses a specific token structure for chat completions (<|start_header_id|> and <|eot_id|>). To prevent hallucinations and get the best performance, you must use the following standard structure:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Sen yardımsever bir Türkçe asistansın.<|eot_id|><|start_header_id|>user<|end_header_id|>

[Write your prompt here]<|eot_id|><|start_header_id|>assistant<|end_header_id|>

🚀 How to Run the Model

You can run this model locally with complete privacy and zero internet connection required.

Option 1: Using Python (llama-cpp-python)

1. Install the library via pip:

pip install llama-cpp-python

2. Download model from Hugging Face:

wget -O Turkish-Llama-3.1-8B-Instruct-q4_k_m.gguf [https://huggingface.co/mehmettozlu/Turkish-Llama-3.1-8B-Instruct-GGUF/resolve/main/Turkish-Llama-3.1-8B-Instruct-q4_k_m.gguf](https://huggingface.co/mehmettozlu/Turkish-Llama-3.1-8B-Instruct-GGUF/resolve/main/Turkish-Llama-3.1-8B-Instruct-q4_k_m.gguf)

3. Create a Python script (run.py):

from llama_cpp import Llama

llm = Llama(
    model_path="./Turkish-Llama-3.1-8B-Instruct-q4_k_m.gguf",
    n_ctx=4096,          # Context window size
    n_gpu_layers=-1      # Offload all layers to GPU
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "Sen yardımsever bir Türkçe asistansın."},
        {"role": "user", "content": "Yapay zeka modellerinin nasıl eğitildiğini kısaca açıklar mısın?"}
    ]
)

print(response["choices"][0]["message"]["content"])

Option 2: Using LM Studio (Easiest - GUI)

• Download and install LM Studio.
• In the search bar, type mehmettozlu/Turkish-Llama-3.1-8B-Instruct-GGUF.
• Download the q4_k_m.gguf file.
• Navigate to the "Chat" tab, ensure the prompt format is set to "Llama 3", and start chatting in Turkish.

Option 3: Using Ollama (For Developers)

Create a text file named Modelfile (no extension) and paste the following content:

FROM hf.co/mehmettozlu/Turkish-Llama-3.1-8B-Instruct-GGUF/Turkish-Llama-3.1-8B-Instruct-q4_k_m.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

"""
SYSTEM """Sen yardımsever bir Türkçe asistansın."""
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|eom_id|>"

Build and run the model:

ollama create turkish-llama-8b -f Modelfile
ollama run turkish-llama-8b