README.md

---
title: "Aqeedah-ao"
emoji: "🕌"
colorFrom: "blue"
colorTo: "green"
sdk: "gradio"
sdk_version: "5.49.1"
app_file: "app.py"
pinned: false
---

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

---
license: mit
task_categories:
- question-answering
- text-retrieval
language:
- ar
tags:
- aqeedah
- islamic-theology
- arabic
- rag
- faiss
- hybrid-search
size_categories:
- n<1K
pretty_name: Aqeedah RAG Dataset
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
dataset_info:
  features:
  - name: content
    dtype: string
  - name: meta
    struct:
    - name: author_name
      dtype: string
    - name: doc_name
      dtype: string
    - name: paragraph_number
      dtype: int64
  - name: embeddings
    list: float64
  splits:
  - name: train
    num_bytes: 43231133
    num_examples: 5419
  download_size: 30017136
  dataset_size: 43231133
---


<div align="center">
  <img src="https://upload.wikimedia.org/wikipedia/en/a/ae/Najran_University_Logo.svg" alt="Najran University Logo" width="200"/>
  
  # Aqeedah AI Assistant �
  
  **A Research Initiative by Najran University, Kingdom of Saudi Arabia**
  
  [![Hugging Face Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-blue)](https://huggingface.co/datasets/abdullah-alamodi/aqeedah-rag-dataset)
  [![Hugging Face Space](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai)
  [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
</div>

---

## 🚀 Quick Start

### Try the Live Demo

**🌐 Live Chatbot:** [https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai](https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai)

### Run Locally

1. **Clone the repository:**
```bash
git clone https://github.com/Abdullah-Alamodi/aqeedah-ai.git
cd aqeedah-ai
```

2. **Install dependencies:**
```bash
pip install -r requirements.txt
```

3. **Set up environment variables:**
Create a `.env` file in the project root:
```env
GEMINI_API_KEY=your_gemini_api_key_here
```

4. **Run the chatbot:**
```bash
python app.py
```

The app will launch at `http://localhost:7860`

### Project Structure

```
aqeedah-ai/
├── app.py                 # Gradio chatbot interface
├── retrieval.py          # Hybrid RAG retrieval system
├── requirements.txt      # Python dependencies
├── rag_playground.ipynb  # Dataset preparation notebook
├── aqeedah_kb.json       # Source data (5419 paragraphs)
├── aqeedah_kb/           # Original source documents (DOCX/PDF)
├── .env                  # API keys (not in git)
├── .gitignore           
├── pyproject.toml        # Project configuration
└── README.md             # This file
```

**Key Features:**
- ✅ **Simple Structure**: Only 2 main Python files (`app.py` + `retrieval.py`)
- ✅ **Cloud-First**: Loads dataset from HuggingFace (no local .bin files)
- ✅ **Production-Ready**: Deployed on HF Spaces with Gradio
- ✅ **Hybrid Retrieval**: BM25 + Dense embeddings (AraBERT)

---

# Aqeedah RAG Dataset 📚

A curated Arabic Islamic theology (Aqeedah) dataset with pre-computed FAISS embeddings, designed for advanced Retrieval-Augmented Generation (RAG) applications in Islamic scholarly research.

## 📋 Dataset Description

This dataset represents a specialized collection of 5419 paragraphs from authoritative Islamic theology texts, meticulously compiled and structured for computational analysis. The corpus focuses specifically on Aqeedah (Islamic creed), covering foundational topics in Islamic belief and theology.

**Key Features:**
- **Authentic Arabic content** with complete diacritics (Tashkeel) preserved for linguistic accuracy
- **Pre-computed semantic embeddings** (768-dimensional dense vectors) using state-of-the-art Arabic language models
- **Rich scholarly metadata** including source document names, author attributions, and precise paragraph references
- **Optimized FAISS index** for millisecond-scale semantic similarity search
- **Hybrid retrieval support** combining traditional keyword-based (BM25) and modern neural approaches

## 🎓 Research Context

This dataset was developed as part of an academic research initiative at **Najran University, Kingdom of Saudi Arabia**, under the supervision of **Dr. Alya Alamodi**, a distinguished scholar holding a Ph.D. in Islamic Theology (Aqeedah). Dr. Alamodi's expertise in classical Islamic sciences combined with modern computational approaches has shaped the careful curation and theological accuracy of this corpus.

The technical implementation and AI infrastructure were designed and developed by **Abdullah Alamodi**, M.Sc. candidate in Artificial Intelligence at **IU International University of Applied Sciences, Germany**. This collaboration represents an interdisciplinary effort bridging traditional Islamic scholarship with cutting-edge natural language processing and information retrieval technologies.

### Research Objectives

1. **Democratizing Access**: Making authoritative Aqeedah knowledge computationally accessible for researchers and students
2. **Semantic Search**: Enabling meaning-based retrieval beyond keyword matching in classical Arabic texts
3. **AI-Assisted Learning**: Supporting intelligent question-answering systems for Islamic education
4. **Scholarly Validation**: Establishing benchmarks for Arabic NLP in religious domain-specific applications

## 📚 Source Texts

This dataset comprises carefully selected paragraphs from the following authoritative Islamic theology works:

1. **شرح الطحاوية** (Sharh al-Tahawiyyah) - صدر الدين محمد بن علاء الدين علي بن محمد ابن أبي العز الحنفي (Volumes 1-2)
2. **كتاب التوحيد** (Kitab al-Tawhid) - محمد بن عبد الوهاب
3. **شرح العقيدة الواسطية** (Sharh al-Aqidah al-Wasitiyyah) - محمد بن صالح بن محمد العثيمين (Volumes 1-2)
4. **القول المفيد على كتاب التوحيد** (Al-Qawl al-Mufid 'ala Kitab al-Tawhid) - محمد بن صالح بن محمد العثيمين (Volumes 1-4)
5. **القول السديد شرح كتاب التوحيد** (Al-Qawl al-Sadid Sharh Kitab al-Tawhid) - عبد الرحمن بن ناصر السعدي
6. **أصول الإيمان** (Usul al-Iman) - عبد العزيز بن عبد الله بن باز
7. **الوجيز في عقيدة السلف الصالح أهل السنة والجماعة** (Al-Wajiz fi Aqidah al-Salaf al-Salih) - عبد الله بن عبد الحميد الأثري
8. **الإسلام أصوله ومبادئه** (Al-Islam: Usuluhu wa Mabadi'uhu) - محمد بن عبد الله بن صالح السحيم
9. **فتاوى نور على الدرب** (Fatawa Nur 'ala al-Darb) - محمد بن صالح بن محمد العثيمين (Volumes 1-4)

## 🗂️ Dataset Structure

### Data Fields

- `paragraph_text` (string): The Arabic text content with complete diacritical marks
- `doc_name` (string): Title of the source Islamic text
- `author_name` (string): Name of the classical or contemporary scholar
- `paragraph_number` (int): Sequential paragraph identifier within the source document
- `embeddings` (list of float): Pre-computed 768-dimensional embedding vector (L2-normalized)

### Data Splits

This dataset contains a single split with 5419 carefully selected paragraphs from verified Islamic theology sources.

## 🤖 Embedding Model

**Model**: [`aubmindlab/bert-base-arabertv02`](https://huggingface.co/aubmindlab/bert-base-arabertv02)

**Technical Specifications**:
- Architecture: BERT-Base (12 layers, 768 hidden dimensions)
- Pre-training: Arabic Wikipedia + other Arabic corpora
- Embedding Dimension: 768
- Text Normalization: Light preprocessing (preserves diacritics for theological accuracy)
- Pooling Strategy: Attention-masked average pooling
- Vector Normalization: L2 normalization for cosine similarity compatibility

## 🚀 Usage

### Installation

```bash
pip install datasets faiss-cpu torch transformers pyarabic rank-bm25
```

### Quick Start

```python
from datasets import load_dataset
import torch
from transformers import AutoTokenizer, AutoModel
import pyarabic.araby as araby

# Load dataset with FAISS index
dataset = load_dataset("abdullah-alamodi/aqeedah-rag-dataset")

# Load the embedding model
model_name = "aubmindlab/bert-base-arabertv02"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Add FAISS index for fast retrieval
dataset['train'].add_faiss_index(column="embeddings")

# Helper function for embedding
def get_embedding(text):
    normalized = araby.normalize_hamza(text)
    text_input = f"query: {normalized}"
    
    inputs = tokenizer([text_input], padding=True, truncation=True, 
                      max_length=512, return_tensors='pt')
    
    with torch.no_grad():
        outputs = model(**inputs)
    
    # Average pooling
    embeddings = outputs.last_hidden_state.mean(dim=1)
    embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    
    return embeddings[0].numpy()

# Search example
query = "ما معنى شهادة أن لا إله إلا الله؟"
query_embedding = get_embedding(query)

# Find top 5 similar documents
scores, retrieved = dataset['train'].get_nearest_examples(
    "embeddings", 
    query_embedding, 
    k=5
)

# Display results
for i, (score, text, meta) in enumerate(zip(
    scores, 
    retrieved['content'],
    retrieved['meta']
)):
    print(f"النص المسترجع للسؤال {i+1}".center(80, '-'))
    print(f"Score: {score:.4f}")
    print(f"Document: {meta['doc_name']} by {meta['author_name']}")
    print(f"Paragraph: {meta['paragraph_number']}")
    print(f"Text: {text[:200]}...")
    print("\n")
```

### Hybrid Search (BM25 + Dense)

```python
from rank_bm25 import BM25Okapi
import numpy as np
import pyarabic.araby as araby
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModel
import torch

# Load dataset with FAISS index
dataset = load_dataset("abdullah-alamodi/aqeedah-rag-dataset")

# Load the embedding model
model_name = "aubmindlab/bert-base-arabertv02"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Add FAISS index for fast retrieval
dataset['train'].add_faiss_index(column="embeddings")

# Helper function for embedding
def get_embedding(text):
    normalized = araby.normalize_hamza(text)
    text_input = f"query: {normalized}"
    
    inputs = tokenizer([text_input], padding=True, truncation=True, 
                      max_length=512, return_tensors='pt')
    
    with torch.no_grad():
        outputs = model(**inputs)
    
    # Average pooling
    embeddings = outputs.last_hidden_state.mean(dim=1)
    embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    
    return embeddings[0].numpy()

# Prepare BM25 index
def normalize_for_bm25(text):
    text = araby.normalize_hamza(text)
    text = araby.strip_diacritics(text)
    text = araby.strip_tatweel(text)
    return text

corpus = [normalize_for_bm25(doc['content']) for doc in dataset['train']]
tokenized = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized)

# Search function
def hybrid_search(query, top_k=5):
    # BM25 search
    norm_query = normalize_for_bm25(query)
    bm25_scores = bm25.get_scores(norm_query.split())
    bm25_top = np.argsort(bm25_scores)[::-1][:top_k]
    
    # Dense search
    query_emb = get_embedding(query)
    scores, faiss_results = dataset['train'].get_nearest_examples(
        "embeddings", query_emb, k=top_k
    )
    
    # Extract FAISS indices (they're already sorted by score)
    # Since get_nearest_examples returns actual data, we need to track indices differently
    # Simple approach: just combine the unique results
    
    # Get unique indices from both methods
    bm25_indices = set(bm25_top.tolist())
    
    # For FAISS, we'll use the returned results directly
    # Combine: prioritize FAISS results, then add BM25-only results
    combined_results = []
    seen_content = set()
    
    # Add FAISS results first
    for content, meta in zip(faiss_results['content'], faiss_results['meta']):
        if content not in seen_content:
            combined_results.append({'content': content, 'meta': meta})
            seen_content.add(content)
    
    # Add unique BM25 results
    for idx in bm25_top:
        doc = dataset['train'][int(idx)]
        if doc['content'] not in seen_content:
            combined_results.append(doc)
            seen_content.add(doc['content'])
            if len(combined_results) >= top_k * 2:  # Get up to 2x results
                break
    
    return combined_results[:top_k * 2]  # Return more results for better coverage

# Example usage
results = hybrid_search("ما هي أركان الإيمان؟", top_k=5)
for i, res in enumerate(results):
    print(f"Result {i+1}: {res['content']}\n")
```

## 📊 Dataset Statistics

- **Total paragraphs**: 5419
- **Language**: Classical and Modern Standard Arabic (ar)
- **Domain**: Islamic Theology (Aqeedah)
- **Source texts**: 17 volumes from 9 distinct scholarly works
- **Average text length**: ~951 characters per paragraph
- **Embedding coverage**: 100% of corpus

## 🎯 Intended Use

### Primary Applications

- ✅ **Scholarly RAG Systems**: Building question-answering systems for Islamic theology education
- ✅ **Semantic Search**: Enabling meaning-based retrieval in classical Arabic religious texts
- ✅ **Educational Technology**: Supporting AI-powered learning platforms for Aqeedah studies
- ✅ **Research Tools**: Facilitating computational analysis of Islamic theological discourse

### Research Domains

- Arabic Natural Language Processing (NLP)
- Information Retrieval in Religious Texts
- Cross-lingual Semantic Search
- Domain-Specific Language Models

---

## 🚀 Deployment to Hugging Face Spaces

### Prerequisites
1. Create a Hugging Face account at [huggingface.co](https://huggingface.co)
2. Get a Gemini API key from [Google AI Studio](https://aistudio.google.com/app/apikey)

### Deployment Steps

1. **Create a new Space:**
   - Go to [huggingface.co/spaces](https://huggingface.co/spaces)
   - Click "Create new Space"
   - Name: `aqeedah-ai`
   - SDK: Select "Gradio"
   - License: MIT

2. **Upload files:**
   ```bash
   # Clone your HF Space
   git clone https://huggingface.co/spaces/abdullah-alamodi/aqeedah-ai
   cd aqeedah-ai
   
   # Copy necessary files
   cp /path/to/aqeedah-ai/app.py .
   cp /path/to/aqeedah-ai/retrieval.py .
   cp /path/to/aqeedah-ai/requirements.txt .
   
   # Commit and push
   git add .
   git commit -m "Initial deployment"
   git push
   ```

3. **Set up secrets:**
   - Go to your Space settings
   - Navigate to "Repository secrets"
   - Add secret: `GEMINI_API_KEY` = your_api_key

4. **Your Space will automatically build and deploy!** 🎉

### Required Files for HF Spaces
- `app.py` - Main Gradio application
- `retrieval.py` - RAG retrieval logic
- `requirements.txt` - Python dependencies

**Note:** The dataset is automatically loaded from HuggingFace, so no need to upload the data files!

---

## ⚠️ Limitations & Considerations

### Scope Limitations
- **Domain Specificity**: Exclusively focused on Islamic theology (Aqeedah); not suitable for general Arabic NLP tasks
- **Language**: Limited to Arabic; no multilingual support
- **Corpus Size**: 5419 paragraphs represent a focused collection, not exhaustive coverage of all Aqeedah literature
- **Temporal Coverage**: Focuses on established scholarly works; may not include the most recent publications

### Theological Considerations
- This dataset is curated for academic and educational purposes
- Users should consult qualified Islamic scholars for authoritative religious guidance
- The dataset represents specific theological perspectives within Sunni Islamic tradition (Ahl al-Sunnah wa al-Jama'ah)

### Technical Limitations
- Embeddings are model-specific (AraBERT v2); transfer to other models may require re-encoding
- FAISS index optimized for CPU inference; GPU acceleration requires additional configuration
- Diacritic preservation may affect compatibility with some NLP tools trained on non-diacritized text

## 📜 License

**MIT License** - This dataset is freely available for academic research, educational purposes, and commercial applications with proper attribution.

## 🙏 Citation

If you use this dataset in your research or applications, please cite:

```bibtex
@dataset{aqeedah_rag_dataset_2025,
  title={Aqeedah RAG Dataset: Arabic Islamic Theology Corpus with Pre-computed Embeddings},
  author={Alamodi, Alya and Alamodi, Abdullah},
  year={2025},
  institution={Najran University, Saudi Arabia},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/datasets/abdullah-alamodi/aqeedah-rag-dataset}},
  note={Curated by Dr. Alya Alamodi (Najran University), Technical Implementation by Abdullah Alamodi (IU International University of Applied Sciences)}
}
```

## 👥 Contributors

**Principal Investigator & Theological Curation:**  
**Dr. Alya Alamodi**  
Ph.D. in Islamic Theology (Aqeedah)  
Najran University, Kingdom of Saudi Arabia

**Technical Development & AI Implementation:**  
**Abdullah Alamodi**  
M.Sc. Candidate in Artificial Intelligence  
IU International University of Applied Sciences, Germany

## 📧 Contact

For questions regarding:
- **Theological content and scholarly interpretation**: Contact Dr. Alya Alamodi via Najran University
- **Technical implementation and AI methodology**: Contact Abdullah Alamodi
- **General inquiries**: Open an issue on the dataset repository

---

**Acknowledgments**: This work was supported by the academic resources of Najran University and developed with computational infrastructure provided by IU International University of Applied Sciences.