First commit

README.md

@@ -1,12 +1,484 @@
 ---
-title: Aqeedah Ai
-emoji: 🐠
-colorFrom: red
-colorTo: pink
-sdk: gradio
-sdk_version: 5.49.1
-app_file: app.py
-pinned: false
+license: mit
+task_categories:
+- question-answering
+- text-retrieval
+language:
+- ar
+tags:
+- aqeedah
+- islamic-theology
+- arabic
+- rag
+- faiss
+- hybrid-search
+size_categories:
+- n<1K
+pretty_name: Aqeedah RAG Dataset
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+dataset_info:
+  features:
+  - name: content
+    dtype: string
+  - name: meta
+    struct:
+    - name: author_name
+      dtype: string
+    - name: doc_name
+      dtype: string
+    - name: paragraph_number
+      dtype: int64
+  - name: embeddings
+    list: float64
+  splits:
+  - name: train
+    num_bytes: 43231133
+    num_examples: 5419
+  download_size: 30017136
+  dataset_size: 43231133
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+<div align="center">
+  <img src="https://upload.wikimedia.org/wikipedia/en/a/ae/Najran_University_Logo.svg" alt="Najran University Logo" width="200"/>
+  
+  # Aqeedah AI Assistant �
+  
+  **A Research Initiative by Najran University, Kingdom of Saudi Arabia**
+  
+  [![Hugging Face Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-blue)](https://huggingface.co/datasets/abdullah-alamodi/aqeedah-rag-dataset)
+  [![Hugging Face Space](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai)
+  [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+</div>
+
+---
+
+## 🚀 Quick Start
+
+### Try the Live Demo
+
+**🌐 Live Chatbot:** [https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai](https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai)
+
+### Run Locally
+
+1. **Clone the repository:**
+```bash
+git clone https://github.com/Abdullah-Alamodi/aqeedah-ai.git
+cd aqeedah-ai
+```
+
+2. **Install dependencies:**
+```bash
+pip install -r requirements.txt
+```
+
+3. **Set up environment variables:**
+Create a `.env` file in the project root:
+```env
+GEMINI_API_KEY=your_gemini_api_key_here
+```
+
+4. **Run the chatbot:**
+```bash
+python app.py
+```
+
+The app will launch at `http://localhost:7860`
+
+### Project Structure
+
+```
+aqeedah-ai/
+├── app.py                 # Gradio chatbot interface
+├── retrieval.py          # Hybrid RAG retrieval system
+├── requirements.txt      # Python dependencies
+├── rag_playground.ipynb  # Dataset preparation notebook
+├── aqeedah_kb.json       # Source data (5419 paragraphs)
+├── aqeedah_kb/           # Original source documents (DOCX/PDF)
+├── .env                  # API keys (not in git)
+├── .gitignore           
+├── pyproject.toml        # Project configuration
+└── README.md             # This file
+```
+
+**Key Features:**
+- ✅ **Simple Structure**: Only 2 main Python files (`app.py` + `retrieval.py`)
+- ✅ **Cloud-First**: Loads dataset from HuggingFace (no local .bin files)
+- ✅ **Production-Ready**: Deployed on HF Spaces with Gradio
+- ✅ **Hybrid Retrieval**: BM25 + Dense embeddings (AraBERT)
+
+---
+
+# Aqeedah RAG Dataset 📚
+
+A curated Arabic Islamic theology (Aqeedah) dataset with pre-computed FAISS embeddings, designed for advanced Retrieval-Augmented Generation (RAG) applications in Islamic scholarly research.
+
+## 📋 Dataset Description
+
+This dataset represents a specialized collection of 5419 paragraphs from authoritative Islamic theology texts, meticulously compiled and structured for computational analysis. The corpus focuses specifically on Aqeedah (Islamic creed), covering foundational topics in Islamic belief and theology.
+
+**Key Features:**
+- **Authentic Arabic content** with complete diacritics (Tashkeel) preserved for linguistic accuracy
+- **Pre-computed semantic embeddings** (768-dimensional dense vectors) using state-of-the-art Arabic language models
+- **Rich scholarly metadata** including source document names, author attributions, and precise paragraph references
+- **Optimized FAISS index** for millisecond-scale semantic similarity search
+- **Hybrid retrieval support** combining traditional keyword-based (BM25) and modern neural approaches
+
+## 🎓 Research Context
+
+This dataset was developed as part of an academic research initiative at **Najran University, Kingdom of Saudi Arabia**, under the supervision of **Dr. Alya Alamodi**, a distinguished scholar holding a Ph.D. in Islamic Theology (Aqeedah). Dr. Alamodi's expertise in classical Islamic sciences combined with modern computational approaches has shaped the careful curation and theological accuracy of this corpus.
+
+The technical implementation and AI infrastructure were designed and developed by **Abdullah Alamodi**, M.Sc. candidate in Artificial Intelligence at **IU International University of Applied Sciences, Germany**. This collaboration represents an interdisciplinary effort bridging traditional Islamic scholarship with cutting-edge natural language processing and information retrieval technologies.
+
+### Research Objectives
+
+1. **Democratizing Access**: Making authoritative Aqeedah knowledge computationally accessible for researchers and students
+2. **Semantic Search**: Enabling meaning-based retrieval beyond keyword matching in classical Arabic texts
+3. **AI-Assisted Learning**: Supporting intelligent question-answering systems for Islamic education
+4. **Scholarly Validation**: Establishing benchmarks for Arabic NLP in religious domain-specific applications
+
+## 📚 Source Texts
+
+This dataset comprises carefully selected paragraphs from the following authoritative Islamic theology works:
+
+1. **شرح الطحاوية** (Sharh al-Tahawiyyah) - صدر الدين محمد بن علاء الدين علي بن محمد ابن أبي العز الحنفي (Volumes 1-2)
+2. **كتاب التوحيد** (Kitab al-Tawhid) - محمد بن عبد الوهاب
+3. **شرح العقيدة الواسطية** (Sharh al-Aqidah al-Wasitiyyah) - محمد بن صالح بن محمد العثيمين (Volumes 1-2)
+4. **القول المفيد على كتاب التوحيد** (Al-Qawl al-Mufid 'ala Kitab al-Tawhid) - محمد بن صالح بن محمد العثيمين (Volumes 1-4)
+5. **القول السديد شرح كتاب التوحيد** (Al-Qawl al-Sadid Sharh Kitab al-Tawhid) - عبد الرحمن بن ناصر السعدي
+6. **أصول الإيمان** (Usul al-Iman) - عبد العزيز بن عبد الله بن باز
+7. **الوجيز في عقيدة السلف الصالح أهل السنة والجماعة** (Al-Wajiz fi Aqidah al-Salaf al-Salih) - عبد الله بن عبد الحميد الأثري
+8. **الإسلام أصوله ومبادئه** (Al-Islam: Usuluhu wa Mabadi'uhu) - محمد بن عبد الله بن صالح السحيم
+9. **فتاوى نور على الدرب** (Fatawa Nur 'ala al-Darb) - محمد بن صالح بن محمد العثيمين (Volumes 1-4)
+
+## 🗂️ Dataset Structure
+
+### Data Fields
+
+- `paragraph_text` (string): The Arabic text content with complete diacritical marks
+- `doc_name` (string): Title of the source Islamic text
+- `author_name` (string): Name of the classical or contemporary scholar
+- `paragraph_number` (int): Sequential paragraph identifier within the source document
+- `embeddings` (list of float): Pre-computed 768-dimensional embedding vector (L2-normalized)
+
+### Data Splits
+
+This dataset contains a single split with 5419 carefully selected paragraphs from verified Islamic theology sources.
+
+## 🤖 Embedding Model
+
+**Model**: [`aubmindlab/bert-base-arabertv02`](https://huggingface.co/aubmindlab/bert-base-arabertv02)
+
+**Technical Specifications**:
+- Architecture: BERT-Base (12 layers, 768 hidden dimensions)
+- Pre-training: Arabic Wikipedia + other Arabic corpora
+- Embedding Dimension: 768
+- Text Normalization: Light preprocessing (preserves diacritics for theological accuracy)
+- Pooling Strategy: Attention-masked average pooling
+- Vector Normalization: L2 normalization for cosine similarity compatibility
+
+## 🚀 Usage
+
+### Installation
+
+```bash
+pip install datasets faiss-cpu torch transformers pyarabic rank-bm25
+```
+
+### Quick Start
+
+```python
+from datasets import load_dataset
+import torch
+from transformers import AutoTokenizer, AutoModel
+import pyarabic.araby as araby
+
+# Load dataset with FAISS index
+dataset = load_dataset("abdullah-alamodi/aqeedah-rag-dataset")
+
+# Load the embedding model
+model_name = "aubmindlab/bert-base-arabertv02"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModel.from_pretrained(model_name)
+
+# Add FAISS index for fast retrieval
+dataset['train'].add_faiss_index(column="embeddings")
+
+# Helper function for embedding
+def get_embedding(text):
+    normalized = araby.normalize_hamza(text)
+    text_input = f"query: {normalized}"
+    
+    inputs = tokenizer([text_input], padding=True, truncation=True, 
+                      max_length=512, return_tensors='pt')
+    
+    with torch.no_grad():
+        outputs = model(**inputs)
+    
+    # Average pooling
+    embeddings = outputs.last_hidden_state.mean(dim=1)
+    embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
+    
+    return embeddings[0].numpy()
+
+# Search example
+query = "ما معنى شهادة أن لا إله إلا الله؟"
+query_embedding = get_embedding(query)
+
+# Find top 5 similar documents
+scores, retrieved = dataset['train'].get_nearest_examples(
+    "embeddings", 
+    query_embedding, 
+    k=5
+)
+
+# Display results
+for i, (score, text, meta) in enumerate(zip(
+    scores, 
+    retrieved['content'],
+    retrieved['meta']
+)):
+    print(f"النص المسترجع للسؤال {i+1}".center(80, '-'))
+    print(f"Score: {score:.4f}")
+    print(f"Document: {meta['doc_name']} by {meta['author_name']}")
+    print(f"Paragraph: {meta['paragraph_number']}")
+    print(f"Text: {text[:200]}...")
+    print("\n")
+```
+
+### Hybrid Search (BM25 + Dense)
+
+```python
+from rank_bm25 import BM25Okapi
+import numpy as np
+import pyarabic.araby as araby
+from datasets import load_dataset
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+# Load dataset with FAISS index
+dataset = load_dataset("abdullah-alamodi/aqeedah-rag-dataset")
+
+# Load the embedding model
+model_name = "aubmindlab/bert-base-arabertv02"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModel.from_pretrained(model_name)
+
+# Add FAISS index for fast retrieval
+dataset['train'].add_faiss_index(column="embeddings")
+
+# Helper function for embedding
+def get_embedding(text):
+    normalized = araby.normalize_hamza(text)
+    text_input = f"query: {normalized}"
+    
+    inputs = tokenizer([text_input], padding=True, truncation=True, 
+                      max_length=512, return_tensors='pt')
+    
+    with torch.no_grad():
+        outputs = model(**inputs)
+    
+    # Average pooling
+    embeddings = outputs.last_hidden_state.mean(dim=1)
+    embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
+    
+    return embeddings[0].numpy()
+
+# Prepare BM25 index
+def normalize_for_bm25(text):
+    text = araby.normalize_hamza(text)
+    text = araby.strip_diacritics(text)
+    text = araby.strip_tatweel(text)
+    return text
+
+corpus = [normalize_for_bm25(doc['content']) for doc in dataset['train']]
+tokenized = [doc.split() for doc in corpus]
+bm25 = BM25Okapi(tokenized)
+
+# Search function
+def hybrid_search(query, top_k=5):
+    # BM25 search
+    norm_query = normalize_for_bm25(query)
+    bm25_scores = bm25.get_scores(norm_query.split())
+    bm25_top = np.argsort(bm25_scores)[::-1][:top_k]
+    
+    # Dense search
+    query_emb = get_embedding(query)
+    scores, faiss_results = dataset['train'].get_nearest_examples(
+        "embeddings", query_emb, k=top_k
+    )
+    
+    # Extract FAISS indices (they're already sorted by score)
+    # Since get_nearest_examples returns actual data, we need to track indices differently
+    # Simple approach: just combine the unique results
+    
+    # Get unique indices from both methods
+    bm25_indices = set(bm25_top.tolist())
+    
+    # For FAISS, we'll use the returned results directly
+    # Combine: prioritize FAISS results, then add BM25-only results
+    combined_results = []
+    seen_content = set()
+    
+    # Add FAISS results first
+    for content, meta in zip(faiss_results['content'], faiss_results['meta']):
+        if content not in seen_content:
+            combined_results.append({'content': content, 'meta': meta})
+            seen_content.add(content)
+    
+    # Add unique BM25 results
+    for idx in bm25_top:
+        doc = dataset['train'][int(idx)]
+        if doc['content'] not in seen_content:
+            combined_results.append(doc)
+            seen_content.add(doc['content'])
+            if len(combined_results) >= top_k * 2:  # Get up to 2x results
+                break
+    
+    return combined_results[:top_k * 2]  # Return more results for better coverage
+
+# Example usage
+results = hybrid_search("ما هي أركان الإيمان؟", top_k=5)
+for i, res in enumerate(results):
+    print(f"Result {i+1}: {res['content']}\n")
+```
+
+## 📊 Dataset Statistics
+
+- **Total paragraphs**: 5419
+- **Language**: Classical and Modern Standard Arabic (ar)
+- **Domain**: Islamic Theology (Aqeedah)
+- **Source texts**: 17 volumes from 9 distinct scholarly works
+- **Average text length**: ~951 characters per paragraph
+- **Embedding coverage**: 100% of corpus
+
+## 🎯 Intended Use
+
+### Primary Applications
+
+- ✅ **Scholarly RAG Systems**: Building question-answering systems for Islamic theology education
+- ✅ **Semantic Search**: Enabling meaning-based retrieval in classical Arabic religious texts
+- ✅ **Educational Technology**: Supporting AI-powered learning platforms for Aqeedah studies
+- ✅ **Research Tools**: Facilitating computational analysis of Islamic theological discourse
+
+### Research Domains
+
+- Arabic Natural Language Processing (NLP)
+- Information Retrieval in Religious Texts
+- Cross-lingual Semantic Search
+- Domain-Specific Language Models
+
+---
+
+## 🚀 Deployment to Hugging Face Spaces
+
+### Prerequisites
+1. Create a Hugging Face account at [huggingface.co](https://huggingface.co)
+2. Get a Gemini API key from [Google AI Studio](https://aistudio.google.com/app/apikey)
+
+### Deployment Steps
+
+1. **Create a new Space:**
+   - Go to [huggingface.co/spaces](https://huggingface.co/spaces)
+   - Click "Create new Space"
+   - Name: `aqeedah-ai`
+   - SDK: Select "Gradio"
+   - License: MIT
+
+2. **Upload files:**
+   ```bash
+   # Clone your HF Space
+   git clone https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai
+   cd aqeedah-ai
+   
+   # Copy necessary files
+   cp /path/to/aqeedah-ai/app.py .
+   cp /path/to/aqeedah-ai/retrieval.py .
+   cp /path/to/aqeedah-ai/requirements.txt .
+   
+   # Commit and push
+   git add .
+   git commit -m "Initial deployment"
+   git push
+   ```
+
+3. **Set up secrets:**
+   - Go to your Space settings
+   - Navigate to "Repository secrets"
+   - Add secret: `GEMINI_API_KEY` = your_api_key
+
+4. **Your Space will automatically build and deploy!** 🎉
+
+### Required Files for HF Spaces
+- `app.py` - Main Gradio application
+- `retrieval.py` - RAG retrieval logic
+- `requirements.txt` - Python dependencies
+
+**Note:** The dataset is automatically loaded from HuggingFace, so no need to upload the data files!
+
+---
+
+## ⚠️ Limitations & Considerations
+
+### Scope Limitations
+- **Domain Specificity**: Exclusively focused on Islamic theology (Aqeedah); not suitable for general Arabic NLP tasks
+- **Language**: Limited to Arabic; no multilingual support
+- **Corpus Size**: 5419 paragraphs represent a focused collection, not exhaustive coverage of all Aqeedah literature
+- **Temporal Coverage**: Focuses on established scholarly works; may not include the most recent publications
+
+### Theological Considerations
+- This dataset is curated for academic and educational purposes
+- Users should consult qualified Islamic scholars for authoritative religious guidance
+- The dataset represents specific theological perspectives within Sunni Islamic tradition (Ahl al-Sunnah wa al-Jama'ah)
+
+### Technical Limitations
+- Embeddings are model-specific (AraBERT v2); transfer to other models may require re-encoding
+- FAISS index optimized for CPU inference; GPU acceleration requires additional configuration
+- Diacritic preservation may affect compatibility with some NLP tools trained on non-diacritized text
+
+## 📜 License
+
+**MIT License** - This dataset is freely available for academic research, educational purposes, and commercial applications with proper attribution.
+
+## 🙏 Citation
+
+If you use this dataset in your research or applications, please cite:
+
+```bibtex
+@dataset{aqeedah_rag_dataset_2025,
+  title={Aqeedah RAG Dataset: Arabic Islamic Theology Corpus with Pre-computed Embeddings},
+  author={Alamodi, Alya and Alamodi, Abdullah},
+  year={2025},
+  institution={Najran University, Saudi Arabia},
+  publisher={Hugging Face},
+  howpublished={\url{https://huggingface.co/datasets/abdullah-alamodi/aqeedah-rag-dataset}},
+  note={Curated by Dr. Alya Alamodi (Najran University), Technical Implementation by Abdullah Alamodi (IU International University of Applied Sciences)}
+}
+```
+
+## 👥 Contributors
+
+**Principal Investigator & Theological Curation:**  
+**Dr. Alya Alamodi**  
+Ph.D. in Islamic Theology (Aqeedah)  
+Najran University, Kingdom of Saudi Arabia
+
+**Technical Development & AI Implementation:**  
+**Abdullah Alamodi**  
+M.Sc. Candidate in Artificial Intelligence  
+IU International University of Applied Sciences, Germany
+
+## 📧 Contact
+
+For questions regarding:
+- **Theological content and scholarly interpretation**: Contact Dr. Alya Alamodi via Najran University
+- **Technical implementation and AI methodology**: Contact Abdullah Alamodi
+- **General inquiries**: Open an issue on the dataset repository
+
+---
+
+**Acknowledgments**: This work was supported by the academic resources of Najran University and developed with computational infrastructure provided by IU International University of Applied Sciences.

app.py

@@ -0,0 +1,364 @@
+"""
+Aqeedah AI Chatbot - Gradio Interface
+======================================
+
+A Retrieval-Augmented Generation (RAG) chatbot for Islamic theology (Aqeedah) 
+questions, powered by Gemini and deployed on Hugging Face Spaces.
+
+Author: Abdullah Alamodi
+Dataset: https://huggingface.co/datasets/abdullah-alamodi/aqeedah-rag-dataset
+Space: https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai
+"""
+
+import os
+import gradio as gr
+from google import genai
+from dotenv import load_dotenv
+from retrieval import HybridRetriever
+
+# Load environment variables
+load_dotenv()
+
+# --- Configuration ---
+GEMINI_API_KEY = os.getenv("GEMINI_API_KEY")
+# Options: "gemini-2.0-flash-exp", "gemini-1.5-flash", "gemini-1.5-pro"
+GEMINI_MODEL = "gemini-2.5-flash"
+
+# System prompt template
+SYSTEM_PROMPT_TEMPLATE = """
+# التعليمات
+1. أجب **فقط** على الأسئلة المتعلقة بـ **العقيدة الإسلامية**.  
+2. لا تستخدم أي معرفة مسبقة خارج السياق.  
+3. إذا لم تجد الإجابة في السياق → قل بوضوح: **"لم أجد الإجابة في النص المقدم"**.  
+4. إذا كان السؤال خارج موضوع العقيدة الإسلامية → قل بوضوح: **"هذا السؤال خارج نطاق العقيدة الإسلامية، ولا أستطيع الإجابة عليه"**.  
+5. في حالة وجود اجابة وكان السؤال متعلقاً بالعقيدة الإسلامية، ابدأ كل إجابة بالصيغة:  
+**"الحمد لله، والصلاة والسلام على رسول الله، وعلى آله وصحبه ومن والاه، أما بعد:"**
+6. اذكر المصادر التي استندت إليها في إجابتك من السياق المقدم مثل اسم الكتاب او اسم الشيخ ان امكن.
+
+---
+
+## السياق:
+{context}
+
+---
+
+## السؤال:
+{query}
+
+---
+## الإجابة:
+الحمد لله، والصلاة والسلام على رسول الله، وعلى آله وصحبه ومن والاه، أما بعد:
+... أكمل اجابتك هنا بناءً على السياق أعلاه.
+"""
+
+# --- Initialize Global Components ---
+print("🚀 Initializing Aqeedah AI Chatbot...")
+
+# Initialize Gemini client
+if not GEMINI_API_KEY:
+    print("⚠️  WARNING: GEMINI_API_KEY not found in environment variables!")
+    print("   Please set GEMINI_API_KEY in .env file or HF Spaces secrets")
+    gemini_client = None
+else:
+    gemini_client = genai.Client(api_key=GEMINI_API_KEY)
+    print(f"✓ Gemini client initialized (model: {GEMINI_MODEL})")
+
+# Initialize retriever
+retriever = HybridRetriever(
+    embedding_model="aubmindlab/bert-base-arabertv02",
+    dataset_name="abdullah-alamodi/aqeedah-rag-dataset"
+)
+
+print("✅ Chatbot initialized successfully!\n")
+
+
+# --- Core Function ---
+
+def answer_question(question, top_k=4):
+    """
+    Answer a question using RAG with Gemini.
+    
+    Args:
+        question (str): User's question in Arabic
+        top_k (int): Number of documents to retrieve
+        
+    Returns:
+        tuple: (answer, sources_list, context_preview)
+    """
+    if not question or question.strip() == "":
+        return "⚠️ الرجاء إدخال سؤال", "", ""
+    
+    # Check if Gemini is available
+    if gemini_client is None:
+        return "⚠️ خطأ: لم يتم تكوين Gemini API. الرجاء التحقق من متغيرات البيئة.", "", ""
+    
+    try:
+        # Step 1: Retrieve relevant documents
+        print(f"\n🔍 Searching for: {question}")
+        results = retriever.search(question, top_k=top_k)
+        
+        if not results:
+            return "⚠️ لم أجد نتائج ذات صلة بسؤالك في قاعدة البيانات", "", ""
+        
+        # Step 2: Build context from retrieved documents
+        context_parts = []
+        sources_list = []
+        
+        for i, result in enumerate(results, 1):
+            doc_name = result['meta']['doc_name']
+            author_name = result['meta']['author_name']
+            paragraph_num = result['meta']['paragraph_number']
+            content = result['content']
+            
+            # Add to context
+            context_parts.append(
+                f"المصدر {i}:\n"
+                f"الكتاب: {doc_name}\n"
+                f"المؤلف: {author_name}\n"
+                f"الفقرة رقم: {paragraph_num}\n"
+                f"النص: {content}\n"
+            )
+            
+            # Add to sources list
+            sources_list.append({
+                "rank": i,
+                "document": doc_name,
+                "author": author_name,
+                "paragraph": paragraph_num,
+                "preview": content[:200] + "..." if len(content) > 200 else content
+            })
+        
+        context = "\n---\n\n".join(context_parts)
+        
+        # Step 3: Generate answer with Gemini
+        print(f"🤖 Generating answer with Gemini...")
+        prompt = SYSTEM_PROMPT_TEMPLATE.format(context=context, query=question)
+        
+        response = gemini_client.models.generate_content(
+            model=GEMINI_MODEL,
+            contents=prompt
+        )
+        
+        answer = response.text
+        
+        # Step 4: Format output
+        print(f"✓ Answer generated successfully")
+        
+        # Create a nicely formatted sources display
+        sources_md = "### 📚 المصادر المستخدمة:\n\n"
+        for src in sources_list:
+            sources_md += f"**{src['rank']}.** {src['document']}\n"
+            sources_md += f"   - المؤلف: {src['author']}\n"
+            sources_md += f"   - الفقرة: {src['paragraph']}\n\n"
+        
+        return answer, sources_md, context
+        
+    except Exception as e:
+        error_msg = f"⚠️ حدث خطأ أثناء معالجة السؤال:\n\n{str(e)}"
+        print(f"❌ Error: {e}")
+        
+        # Check if it's a rate limit error
+        if "429" in str(e) or "RESOURCE_EXHAUSTED" in str(e):
+            error_msg = """## ⚠️ تم تجاوز الحد المسموح من الطلبات (API Rate Limit Exceeded)
+
+**عذراً! لقد تم استنفاد حصة Gemini API المجانية.**
+
+### 🔄 الحلول المتاحة:
+
+**الحل الأول: انتظر دقيقة واحدة ثم أعد المحاولة**
+- انتظر 60 ثانية
+- أعد إدخال السؤال مرة أخرى
+
+**الحل الثاني: قم بتنزيل الكود وتشغيله محلياً**
+
+1. قم بتحميل الكود من GitHub:
+   ```bash
+   git clone https://github.com/Abdullah-Alamodi/aqeedah-ai.git
+   cd aqeedah-ai
+   ```
+
+2. قم بإنشاء ملف `.env` وأضف مفتاح Gemini API الخاص بك:
+   ```
+   GEMINI_API_KEY=your_api_key_here
+   ```
+   احصل على المفتاح من: [Google AI Studio](https://aistudio.google.com/app/apikey)
+
+3. قم بتثبيت المتطلبات:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+4. شغّل التطبيق:
+   ```bash
+   python app.py
+   ```
+
+---
+
+**📊 GitHub Repository:** [Abdullah-Alamodi/aqeedah-ai](https://github.com/Abdullah-Alamodi/aqeedah-ai)
+
+**💡 ملاحظة:** التشغيل المحلي يعطيك حصة API مستقلة وأداء أفضل!
+"""
+        
+        return error_msg, "", ""
+
+
+# --- Gradio Interface ---
+
+def create_interface():
+    """Create and configure the Gradio interface."""
+    
+    with gr.Blocks(
+        title="مساعد العقيدة الإسلامية | Aqeedah AI Assistant",
+        theme=gr.themes.Soft(),
+        css="""
+        .rtl { direction: rtl; text-align: right; }
+        .container { max-width: 900px; margin: auto; }
+        """
+    ) as interface:
+        
+        # Header
+        gr.Markdown("""
+        <div align="center">
+            <img src="https://upload.wikimedia.org/wikipedia/en/a/ae/Najran_University_Logo.svg" alt="Najran University Logo" width="200">
+        </div>
+                    
+        <div style="text-align: center;">
+            <h1>Aqeedah-Ai 🕌</h1>
+            <p>نظام ذكاء اصطناعي للإجابة على أسئلة العقيدة الإسلامية بناءً على مصادر علمية موثوقة</p>
+        </div>
+        
+        ---
+        
+        <div style="text-align: center; padding: 15px; background-color: rgba(255, 243, 205, 0.3); border: 2px solid #ff9800; border-radius: 8px; margin: 10px 0;">
+            <h3 style="color: #ff6b6b; margin-top: 0;">⚠️ تحذير هام | Important Warning</h3>
+            <p style="margin: 10px 0; line-height: 1.8;">
+                <strong>🔬 هذا نظام تجريبي للبحث العلمي فقط</strong><br>
+                <strong>🔬 This is an Experimental Research Chatbot</strong>
+            </p>
+            <p style="margin: 10px 0; line-height: 1.8; direction: rtl; text-align: center;">
+                ⚠️ <strong>لا تعتمد على إجابات هذا النظام في الأمور الشرعية</strong><br>
+                يجب الرجوع إلى العلماء المعتمدين (المشايخ) للفتاوى والأحكام الشرعية
+            </p>
+            <p style="margin: 10px 0; line-height: 1.8;">
+                ⚠️ <strong>Do not rely on this system for religious rulings</strong><br>
+                Always refer to authorized Islamic scholars (Sheikh) for religious guidance
+            </p>
+        </div>
+
+        """)
+        
+        gr.Markdown("### 💬 اطرح سؤالك")
+        
+        with gr.Row():
+            with gr.Column(scale=4):
+                question_input = gr.Textbox(
+                    label="السؤال (Question)",
+                    placeholder="مثال: ما معنى شهادة أن لا إله إلا الله؟",
+                    lines=3,
+                    elem_classes=["rtl"]
+                )
+            with gr.Column(scale=1):
+                top_k_slider = gr.Slider(
+                    minimum=2,
+                    maximum=8,
+                    value=4,
+                    step=1,
+                    label="عدد المصادر (Sources)",
+                    info="Number of documents to retrieve"
+                )
+        
+        submit_btn = gr.Button("🔍 ابحث واجب | Search & Answer", variant="primary", size="lg")
+        
+        gr.Markdown("### 📝 الإجابة (Answer)")
+        answer_output = gr.Textbox(
+            label="",
+            lines=10,
+            elem_classes=["rtl"],
+            show_label=False
+        )
+        
+        gr.Markdown("### 📚 المصادر المستخدمة (Sources)")
+        sources_output = gr.Markdown(
+            value="",
+            elem_classes=["rtl"]
+        )
+        
+        # Advanced: Show context (collapsible)
+        with gr.Accordion("🔍 عرض السياق الكامل (Show Full Context)", open=False):
+            context_output = gr.Textbox(
+                label="السياق المستخدم",
+                lines=15,
+                elem_classes=["rtl"]
+            )
+        
+        # Example questions
+        gr.Markdown("### 💡 أمثلة للأسئلة (Example Questions)")
+        gr.Examples(
+            examples=[
+                ["ما معنى شهادة أن لا إله إلا الله وأن محمدا رسول الله صلى الله عليه وسلم؟"],
+                ["ما الفرق بين الإسلام والإيمان؟"],
+                ["مامعنى قَوْلُهُ: (لَهُ مَعْنَى الرُّبُوبِيَّةِ وَلَا مَرْبُوبَ، وَمَعْنَى الْخَالِقِ وَلَا مَخْلُوقَ)؟"],
+                ["مالمقصود بالايمان بالقدر خيره وشره؟"],
+                ["مالمقصود بالنشرة؟"],
+                ["ما أقسام سب الدهر؟"],
+                ["ماحكم قول: ما شاء الله وشئت؟"],
+                ["ما صفات وميزات أهل السنة والجماعة؟"],
+                ["كيف تحقق محبة الرسول صلى الله عليه وسلم؟"],
+                ["من هي الفرقة الناجية وما صفاتها وما أبرز خصائصها؟"]
+            ],
+            inputs=question_input,
+            label=""
+        )
+        
+        # Footer
+        gr.Markdown("""
+        ---
+        <div style="text-align: center; padding: 10px; background-color: rgba(100, 100, 100, 0.1); border-radius: 5px; margin: 10px 0;">
+            <strong>🎓 Researcher name | اسم الباحثة:</strong><br>
+            <strong>Dr. Alya Alamodi</strong> - Ph.D. in Islamic Theology (Aqeedah)<br>
+            Najran University, Kingdom of Saudi Arabia<br><br>
+            <strong>🤖 AI Architecture & Development:</strong> <em>Abdullah Alamodi, M.Sc. AI (IU International University, Germany)</em>
+        </div>
+        
+        <div style="text-align: center; font-size: 0.9em; color: #666;">
+            <p>📊 <strong>Dataset:</strong> <a href="https://huggingface.co/datasets/abdullah-alamodi/aqeedah-rag-dataset" target="_blank">abdullah-alamodi/aqeedah-rag-dataset</a></p>
+            <p>🤖 <strong>Model:</strong> Gemini 2.5 Flash + AraBERT v2</p>
+            <p>⚠️ <strong>Disclaimer:</strong> This is an AI research tool. Please consult qualified Islamic scholars for authoritative religious guidance.</p>
+            <p>📜 <strong>License:</strong> MIT | <strong>Sources:</strong> Authenticated Sunni Islamic theology texts</p>
+        </div>
+        """)
+        
+        # Connect components
+        submit_btn.click(
+            fn=answer_question,
+            inputs=[question_input, top_k_slider],
+            outputs=[answer_output, sources_output, context_output]
+        )
+        
+        # Also trigger on Enter key
+        question_input.submit(
+            fn=answer_question,
+            inputs=[question_input, top_k_slider],
+            outputs=[answer_output, sources_output, context_output]
+        )
+    
+    return interface
+
+
+# --- Launch App ---
+
+if __name__ == "__main__":
+    app = create_interface()
+    
+    # Launch configuration
+    # Use localhost for local development, 0.0.0.0 for deployment
+    is_hf_space = os.getenv("SPACE_ID") is not None
+    
+    app.launch(
+        server_name="0.0.0.0" if is_hf_space else "127.0.0.1",
+        server_port=7860,
+        share=False,
+        show_error=True,
+        inbrowser=True  # Auto-open browser on local
+    )

requirements.txt

@@ -0,0 +1,123 @@
+aiofiles==24.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.2
+aiosignal==1.4.0
+annotated-doc==0.0.3
+annotated-types==0.7.0
+anyio==4.11.0
+asttokens==3.0.0
+attrs==25.4.0
+audioop-lts==0.2.2
+brotli==1.1.0
+cachetools==6.2.1
+certifi==2025.10.5
+charset-normalizer==3.4.4
+click==8.3.0
+colorama==0.4.6
+comm==0.2.3
+datasets==4.3.0
+debugpy==1.8.17
+decorator==5.2.1
+dill==0.4.0
+executing==2.2.1
+faiss-cpu==1.12.0
+fastapi==0.120.4
+ffmpy==0.6.4
+filelock==3.20.0
+frozenlist==1.8.0
+fsspec==2025.9.0
+google-auth==2.42.0
+google-genai==1.47.0
+gradio==5.49.1
+gradio-client==1.13.3
+groovy==0.1.2
+h11==0.16.0
+hf-xet==1.2.0
+httpcore==1.0.9
+httpx==0.28.1
+huggingface-hub==0.36.0
+idna==3.11
+ipykernel==7.1.0
+ipython==9.6.0
+ipython-pygments-lexers==1.1.1
+jedi==0.19.2
+jinja2==3.1.6
+joblib==1.5.2
+jupyter-client==8.6.3
+jupyter-core==5.9.1
+lxml==6.0.2
+markdown-it-py==4.0.0
+markupsafe==3.0.3
+matplotlib-inline==0.2.1
+mdurl==0.1.2
+mpmath==1.3.0
+multidict==6.7.0
+multiprocess==0.70.16
+nest-asyncio==1.6.0
+networkx==3.5
+nltk==3.9.2
+numpy==2.3.4
+orjson==3.11.4
+packaging==25.0
+pandas==2.3.3
+parso==0.8.5
+pillow==11.3.0
+platformdirs==4.5.0
+prompt-toolkit==3.0.52
+propcache==0.4.1
+psutil==7.1.2
+pure-eval==0.2.3
+pyarabic==0.6.15
+pyarrow==22.0.0
+pyasn1==0.6.1
+pyasn1-modules==0.4.2
+pydantic==2.11.10
+pydantic-core==2.33.2
+pydub==0.25.1
+pygments==2.19.2
+python-dateutil==2.9.0.post0
+python-docx==1.2.0
+python-dotenv==1.2.1
+python-multipart==0.0.20
+pytz==2025.2
+pyyaml==6.0.3
+pyzmq==27.1.0
+rank-bm25==0.2.2
+regex==2025.10.23
+requests==2.32.5
+rich==14.2.0
+rsa==4.9.1
+ruff==0.14.3
+safehttpx==0.1.7
+safetensors==0.6.2
+scikit-learn==1.7.2
+scipy==1.16.3
+semantic-version==2.10.0
+sentence-transformers==5.1.2
+setuptools==80.9.0
+shellingham==1.5.4
+six==1.17.0
+sniffio==1.3.1
+stack-data==0.6.3
+starlette==0.49.1
+sympy==1.14.0
+tenacity==9.1.2
+threadpoolctl==3.6.0
+tokenizers==0.22.1
+tomlkit==0.13.3
+torch==2.9.0
+torchvision==0.24.0
+tornado==6.5.2
+tqdm==4.67.1
+traitlets==5.14.3
+transformers==4.57.1
+typer==0.20.0
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzdata==2025.2
+urllib3==2.5.0
+uvicorn==0.38.0
+wcwidth==0.2.14
+websockets==15.0.1
+xxhash==3.6.0
+yarl==1.22.0

retrieval.py

@@ -0,0 +1,237 @@
+"""
+Aqeedah RAG Retrieval System
+============================
+
+This module implements a hybrid retrieval system combining BM25 (sparse) and 
+dense embeddings (FAISS) for Arabic Islamic theology (Aqeedah) question-answering.
+
+The system loads pre-computed embeddings from HuggingFace and performs efficient
+semantic search over a corpus of authenticated Islamic scholarly texts.
+
+Author: Abdullah Alamodi
+Dataset: https://huggingface.co/datasets/abdullah-alamodi/aqeedah-rag-dataset
+"""
+
+import numpy as np
+import torch
+import pyarabic.araby as araby
+from rank_bm25 import BM25Okapi
+from transformers import AutoTokenizer, AutoModel
+from datasets import load_dataset
+from tqdm import tqdm
+
+
+# --- Normalization Functions ---
+
+def normalize_for_bm25(text):
+    """
+    Aggressive normalization for keyword matching.
+    
+    Args:
+        text (str): Arabic text to normalize
+        
+    Returns:
+        str: Normalized text suitable for BM25 indexing
+    """
+    text = araby.normalize_hamza(text)   # Normalize alef, hamza
+    text = araby.strip_diacritics(text)  # Strip all diacritics
+    text = araby.strip_tatweel(text)     # Strip tatweel (elongation)
+    return text
+
+
+def normalize_for_dense(text):
+    """
+    Light normalization for dense embeddings, preserves diacritics.
+    
+    Args:
+        text (str): Arabic text to normalize
+        
+    Returns:
+        str: Lightly normalized text suitable for dense embeddings
+    """
+    text = araby.normalize_hamza(text)   # Only normalize alef, hamza
+    return text
+
+
+# --- Helper function for pooling embeddings ---
+
+def _average_pool(last_hidden_states, attention_mask):
+    """
+    Performs average pooling on the last hidden states, respecting the attention mask.
+    
+    Args:
+        last_hidden_states: Model output hidden states
+        attention_mask: Attention mask tensor
+        
+    Returns:
+        torch.Tensor: Pooled embeddings
+    """
+    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
+    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
+
+
+# --- HybridRetriever Class ---
+
+class HybridRetriever:
+    """
+    Hybrid retrieval system combining BM25 and dense embeddings.
+    
+    This class loads the Aqeedah dataset from HuggingFace, creates a BM25 index,
+    and loads pre-computed FAISS embeddings for efficient hybrid search.
+    
+    Args:
+        embedding_model (str): HuggingFace model name for query encoding
+        dataset_name (str): HuggingFace dataset name
+        batch_size (int): Batch size for encoding (if needed)
+        use_gpu (bool): Whether to use GPU if available
+    """
+    
+    def __init__(
+        self, 
+        embedding_model="aubmindlab/bert-base-arabertv02",
+        dataset_name="abdullah-alamodi/aqeedah-rag-dataset",
+        batch_size=32,
+        use_gpu=True
+    ):
+        self.dataset_name = dataset_name
+        self.batch_size = batch_size
+        self.device = torch.device("cuda" if (torch.cuda.is_available() and use_gpu) else "cpu")
+        
+        print(f"🚀 Initializing HybridRetriever")
+        print(f"   Device: {self.device}")
+        print(f"   Embedding model: {embedding_model}")
+        print(f"   Dataset: {dataset_name}")
+        
+        # --- Load Dataset from HuggingFace ---
+        print(f"\n📥 Loading dataset from HuggingFace...")
+        self.dataset = load_dataset(dataset_name, split="train")
+        self.documents = [
+            {
+                "content": doc["content"],
+                "meta": doc["meta"]
+            }
+            for doc in self.dataset
+        ]
+        print(f"   ✓ Loaded {len(self.documents)} documents")
+        
+        # --- Load Dense Model ---
+        print(f"\n🤖 Loading embedding model...")
+        self.dense_tokenizer = AutoTokenizer.from_pretrained(embedding_model)
+        self.dense_model = AutoModel.from_pretrained(embedding_model).to(self.device).eval()
+        print(f"   ✓ Model loaded on {self.device}")
+        
+        # --- Create BM25 Index ---
+        print(f"\n🔍 Building BM25 index...")
+        bm25_corpus = [normalize_for_bm25(doc['content']) for doc in self.documents]
+        tokenized_corpus = [doc.split() for doc in tqdm(bm25_corpus, desc="   Tokenizing")]
+        self.bm25 = BM25Okapi(tokenized_corpus)
+        print(f"   ✓ BM25 index created")
+        
+        # --- Load FAISS Index from HuggingFace ---
+        print(f"\n⚡ Loading FAISS index...")
+        try:
+            # Try to load pre-computed index from dataset
+            self.dataset.load_faiss_index("embeddings", "embeddings_index")
+            print(f"   ✓ FAISS index loaded from HuggingFace")
+        except Exception as e:
+            print(f"   ⚠️  Could not load FAISS index from HF: {e}")
+            print(f"   📊 Creating FAISS index from embeddings column...")
+            # Create index from embeddings column if not available
+            self.dataset.add_faiss_index(column="embeddings")
+            print(f"   ✓ FAISS index created")
+        
+        print(f"\n✅ HybridRetriever initialized successfully!\n")
+    
+    def search(self, query, top_k=5):
+        """
+        Perform hybrid search combining BM25 and dense retrieval.
+        
+        Args:
+            query (str): Arabic query text
+            top_k (int): Number of top results to return
+            
+        Returns:
+            list: List of retrieved documents with metadata
+        """
+        # A. Sparse Search (BM25)
+        norm_query_bm25 = normalize_for_bm25(query)
+        tokenized_query = norm_query_bm25.split()
+        bm25_scores = self.bm25.get_scores(tokenized_query)
+        bm25_top_k_indices = np.argsort(bm25_scores)[::-1][:top_k]
+        
+        # B. Dense Search (FAISS)
+        norm_query_dense = f"query: {normalize_for_dense(query)}"
+        inputs = self.dense_tokenizer(
+            [norm_query_dense], 
+            max_length=512, 
+            padding=True, 
+            truncation=True, 
+            return_tensors='pt'
+        ).to(self.device)
+        
+        with torch.no_grad():
+            outputs = self.dense_model(**inputs)
+        
+        query_embedding = _average_pool(outputs.last_hidden_state, inputs['attention_mask'])
+        query_embedding = torch.nn.functional.normalize(query_embedding, p=2, dim=1)
+        query_embedding = query_embedding.cpu().numpy()[0]
+        
+        # Search using HuggingFace dataset's FAISS index
+        scores, retrieved = self.dataset.get_nearest_examples(
+            "embeddings", 
+            query_embedding, 
+            k=top_k
+        )
+        
+        # Get FAISS indices (retrieve actual indices from the dataset)
+        # Since get_nearest_examples returns the actual data, we need to find indices
+        faiss_contents = retrieved['content']
+        faiss_indices = []
+        for content in faiss_contents:
+            # Find index of this content in original documents
+            for idx, doc in enumerate(self.documents):
+                if doc['content'] == content:
+                    faiss_indices.append(idx)
+                    break
+        
+        # C. Combine results (union of BM25 and FAISS)
+        fused_indices = list(set(bm25_top_k_indices.tolist()) | set(faiss_indices))
+        
+        # Return unique results
+        results = []
+        seen_contents = set()
+        for idx in fused_indices:
+            if idx < len(self.documents):
+                content = self.documents[idx]['content']
+                if content not in seen_contents:
+                    results.append(self.documents[idx])
+                    seen_contents.add(content)
+        
+        return results[:top_k * 2]  # Return up to 2x top_k for better coverage
+
+
+# --- Example Usage ---
+
+if __name__ == "__main__":
+    # Initialize retriever
+    retriever = HybridRetriever()
+    
+    # Example queries
+    test_queries = [
+        "ما معنى شهادة أن لا إله إلا الله وأن محمدا رسول الله صلى الله عليه وسلم؟",
+        "ما الفرق بين الإسلام والإيمان؟",
+        "ما هي أركان الإيمان؟"
+    ]
+    
+    for query in test_queries:
+        print(f"\n{'='*80}")
+        print(f"❓ Query: {query}")
+        print(f"{'='*80}")
+        
+        results = retriever.search(query, top_k=3)
+        
+        for i, result in enumerate(results, 1):
+            print(f"\n[{i}] {result['meta']['doc_name']}")
+            print(f"    Author: {result['meta']['author_name']}")
+            print(f"    Paragraph: {result['meta']['paragraph_number']}")
+            print(f"    Preview: {result['content'][:150]}...")