yemeni-ai-lab/gemma-4-e2b-yemeni-arabic-assistant-lora

Gemma 4 E2B Yemeni Arabic Assistant LoRA

gemma-4-e2b-yemeni-arabic-assistant-lora is a LoRA adapter for Gemma 4 E2B Instruct, adapted for responses in broad Yemeni Arabic.

The goal of this adapter is not to create a new general-purpose model from scratch. It is a dialect adaptation experiment: it pushes a capable Arabic/English instruction model toward Yemeni-style wording, shorter local responses, Yemeni dialect rewriting, and translation into Yemeni Arabic while preserving the base model's general instruction-following ability as much as possible.

Yemeni Arabic is underrepresented in public LLM resources. This release is intended to support research and practical experimentation around low-resource Arabic dialect adaptation, especially for Yemeni Arabic assistant responses.

To the best of our knowledge, this is the first publicly released model focused specifically on Yemeni Arabic assistant-style responses.

Model Details

• Model name: gemma-4-e2b-yemeni-arabic-assistant-lora
• Base model: unsloth/gemma-4-E2B-it
• Developer: Loay Amin
• Organization: Yemeni AI Lab
• Training framework: Unsloth
• Fine-tuning method: QLoRA + LoRA
• Language: Arabic
• Dialect focus: Broad Yemeni Arabic
• Task type: Text generation / conversational assistant / dialect rewriting
• Adapter format: PEFT LoRA

Intended Use

This adapter is intended for:

• Yemeni Arabic assistant responses
• Yemeni dialect rewriting
• Translation or adaptation into Yemeni Arabic
• Short localized Arabic assistant replies
• Yemeni vocabulary and expression experiments
• Research on low-resource Arabic dialect fine-tuning
• Evaluation of synthetic and grounded data for dialect adaptation

Example use cases:

User: ترجم للهجة اليمنية: I am tired today and I do not want to go out.
Assistant: أنا تعبان اليوم وما اشتي أخرج.

User: اكتبها بطريقة يمنية أبسط: لا أريد أن أتكلم عن الموضوع الآن.
Assistant: ما اشتي أتكلم عن الموضوع ذلحين.

User: رد على: وينك مختفي؟
Assistant: موجود والله بس اليوم كان فوق رأسي شغل.

Training Data

The adapter was trained on 19,609 cleaned Yemeni Arabic assistant-style samples.

The dataset was built from a mixture of locally collected and generated resources, including:

• Yemeni social-media style text collected from public/community sources
• Yemeni examples derived from Arabic dialect identification data
• Yemeni vocabulary and expression references
• Local dictionary-style resources
• Grounded teacher-model rewrites and expansions
• Filtered task examples for translation, rewriting, Q&A, reasoning, and calculations

The final training file was cleaned with rule-based and LLM-assisted review passes. The cleaning focused on removing or reducing:

• Non-target dialect leakage such as Gulf/Egyptian expressions
• Overused assistant phrases and service endings
• Excessive greetings and formal closings
• Repetitive openings such as overuse of "والله" and "شوف"
• Long, generic explanations where a direct answer was better
• Obvious hallucinated or malformed Yemeni expressions
• Known unwanted terms from the current data cleaning pass

The final cleaned dataset used for this run contained:

• 19,609 rows
• Average assistant response length around 150 characters
• Maximum assistant response length capped at 360 characters

Training Configuration

The model was trained for 1 epoch using Unsloth.

Parameter	Value
Method	QLoRA
Quantization	4-bit
Epochs	1
Training samples	19,609
Steps	2,436
Context length	1,024
Batch size	2
Learning rate	0.0002
Warmup steps	50
Optimizer	AdamW 8-bit
LoRA rank	32
LoRA alpha	64
LoRA dropout	0
Final training loss	2.4486

Recommended System Prompt

For best results, use a short system prompt that explicitly controls assistant behavior:

أنت مساعد يمني ذكي ومختصر.
جاوب باللهجة اليمنية الطبيعية.
لا تضف تحية إلا إذا المستخدم بدأ بتحية.
لا تختم بسؤال أو عرض مساعدة.
لا تضف رأيك إلا إذا المستخدم طلب رأيك.
في الترجمة أو إعادة الصياغة: أعطِ النص فقط بدون شرح.

For translation-only tasks:

ترجم للهجة اليمنية فقط. لا تضف مقدمة ولا شرح.

Recommended Inference Settings

Suggested generation settings:

temperature: 0.3 - 0.5
top_p: 0.85 - 0.9
top_k: 40
repetition_penalty: 1.08 - 1.12
presence_penalty: 0
max_new_tokens: 160 - 256

For translation or rewriting tasks, use lower temperature:

temperature: 0.2 - 0.3
max_new_tokens: 120 - 180

If the model becomes too chatty, reduce max_new_tokens first. Many assistant-style continuation problems come from allowing the model to generate far past the natural answer length.

Usage with PEFT

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = "unsloth/gemma-4-E2B-it"
adapter = "yemeni-ai-lab/gemma-4-e2b-yemeni-arabic-assistant-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

model = PeftModel.from_pretrained(model, adapter)

messages = [
    {
        "role": "system",
        "content": "أنت مساعد يمني ذكي ومختصر. جاوب باللهجة اليمنية الطبيعية."
    },
    {
        "role": "user",
        "content": "كيف حالك اليوم؟"
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=160,
    temperature=0.4,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Data and Dialect Notes

Yemeni Arabic is not one dialect. It includes substantial regional variation across Sana'a, Taiz, Aden, Ibb, Hadramout, Tihama, Dhamar, Lahj, Al-Bayda, Marib, and other areas. This adapter targets a broad, practical Yemeni style for assistant use rather than a single city-specific dialect.

The dataset intentionally avoids some expressions that appeared in generated data but were judged unsuitable for the target style, including examples such as non-target vocatives, Gulf-like fillers, and malformed Yemeni phrases.

Limitations

This is an experimental dialect fine-tune.

Known limitations:

• The adapter may still produce Modern Standard Arabic on technical, formal, or unfamiliar topics.
• The base instruct model may still show polite assistant behavior unless the system prompt and decoding length are controlled.
• The model may occasionally overuse common openings such as "والله" or "شوف".
• It may hallucinate factual or cultural information if used without retrieval or source grounding.
• The training data includes synthetic and teacher-assisted samples, so not every phrase should be treated as corpus-attested Yemeni speech.
• It is not a legal, medical, financial, or religious authority.
• It does not represent every Yemeni region equally.
• It may still require prompt and decoding control for very casual social conversations.

Ethical and Privacy Considerations

The training work used collected and generated Arabic text for dialect adaptation. Public or community-sourced social-media text can contain names, opinions, jokes, and local references. Cleaning passes were applied to reduce low-quality generations and unwanted assistant behavior, but users should not assume the data pipeline guarantees perfect anonymization or factual accuracy.

This model should not be used to impersonate private individuals, generate harassment, or make high-stakes decisions about people.

Citation

If you use this model, cite it as:

@misc{amin2026gemma4e2byemeniarabicassistant,
  title={Gemma 4 E2B Yemeni Arabic Assistant LoRA},
  author={Amin, Loay},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/yemeni-ai-lab/gemma-4-e2b-yemeni-arabic-assistant-lora}}
}

Developer

Developed by Loay Amin under Yemeni AI Lab.

This project is part of an effort to improve AI support for Yemeni Arabic.