# Guide: Updating Your RAG System Content

This guide explains how to clear, update, and add new content to your vector database.

## Table of Contents
1. [Clearing the Vector Database](#clearing-the-vector-database)
2. [Adding New Content](#adding-new-content)
3. [Adding New Repositories](#adding-new-repositories)
4. [Adding Portfolio Website Data](#adding-portfolio-website-data)
5. [Rebuilding the Index](#rebuilding-the-index)

---

## Clearing the Vector Database

To start fresh with new content, you need to delete the existing vector database files:

### Method 1: Manual Deletion
Delete these files in `docs/faiss/`:
- `index.faiss` - The FAISS vector index
- `index.pkl` - Document store and index mapping
- `metadata.pkl` - Metadata for all documents
- `document_lookup.txt` - Human-readable document index

### Method 2: Using Python
```python
import os
import shutil

faiss_dir = "docs/faiss"
if os.path.exists(faiss_dir):
    for file in ["index.faiss", "index.pkl", "metadata.pkl", "document_lookup.txt"]:
        filepath = os.path.join(faiss_dir, file)
        if os.path.exists(filepath):
            os.remove(filepath)
            print(f"Deleted {file}")
```

---

## Adding New Content

### 1. Adding PDF Documents

**Location**: `docs/pdfs/`

**Steps**:
1. Copy your PDF files to `docs/pdfs/`
2. Supported formats: `.pdf`
3. The system automatically extracts text from all PDFs in this directory
4. Each page is processed separately for better retrieval

**Example**:
```bash
# Copy new PDFs
cp /path/to/your/resume.pdf docs/pdfs/
cp /path/to/your/paper.pdf docs/pdfs/
```

### 2. Adding Audio Transcripts

**Location**: `docs/youtube/`

**Steps**:
1. Place transcript files in `docs/youtube/`
2. Naming convention: `*_transcript.txt`
3. The system automatically processes all files matching this pattern

**Example**:
```bash
# Add transcript
cp interview_transcript.txt docs/youtube/MyInterview_transcript.txt
```

### 3. Adding README Files Manually

**Location**: `docs/readmes/`

**Steps**:
1. Copy README files to `docs/readmes/`
2. Naming convention: `*-README.md` (recommended)
3. Files containing "Tpoze-subnet" (case-insensitive) are automatically excluded

**Example**:
```bash
# Add a README manually
cp /path/to/project/README.md docs/readmes/MyProject-README.md
```

---

## Adding New Repositories

### Method 1: Using the Download Script (Recommended)

The `download_readmes.py` script automatically downloads READMEs from your GitHub repositories.

**Steps**:
1. Get a GitHub Personal Access Token (optional but recommended):
   - Go to GitHub Settings → Developer settings → Personal access tokens
   - Create a token with `public_repo` scope
   - Copy the token

2. Run the download script:
```bash
# Basic usage (uses public API, rate-limited)
python download_readmes.py --username DaJulster

# With token (higher rate limits)
python download_readmes.py --username DaJulster --token YOUR_GITHUB_TOKEN

# Show available repos without downloading
python download_readmes.py --username DaJulster --show-available

# Filter options
python download_readmes.py --username DaJulster \
    --exclude-forks \
    --min-stars 1 \
    --exclude-tpoze \
    --contribution-type owner
```

**Script Options**:
- `--username`: Your GitHub username (required)
- `--token`: GitHub personal access token (optional, increases rate limit)
- `--output-dir`: Output directory (default: `docs/readmes`)
- `--exclude-forks`: Exclude forked repositories
- `--min-stars`: Minimum stars required
- `--exclude-tpoze`: Exclude repos containing 'tpoze' in name
- `--contribution-type`: Filter by `owner`, `contributor`, `committer`, or `all`
- `--show-available`: Show repos without downloading

**What the Script Does**:
- Finds all repositories you own
- Finds repositories you've contributed to (commits, PRs, issues)
- Checks organizations you're a member of
- Downloads README files from each repository
- Saves them with metadata to `docs/readmes/`

### Method 2: Manual Addition

1. Clone or download the repository
2. Copy the README file:
```bash
cp /path/to/repo/README.md docs/readmes/RepoName-README.md
```

3. Optionally add metadata at the top of the file:
```markdown
# Repository Name
**Repository:** owner/repo-name
**Contribution:** 👑 Owner
**Stars:** 42
**Description:** Project description
**Language:** Python
**Last Updated:** 2024-01-01
---
```

---

## Adding Portfolio Website Data

The portfolio website is automatically scraped during indexing. The system:

1. **Scrapes the main page**: `https://julien-ser.github.io/JulienSerbanescu/`
2. **Follows internal links**: Automatically discovers and scrapes linked pages (up to 5 pages)
3. **Extracts main content**: Focuses on main content areas, ignoring navigation/footers
4. **Handles dynamic content**: Removes scripts and styles for clean text extraction

**To change the portfolio URL**, edit `build_index.py`:
```python
portfolio_url = "https://your-portfolio-url.com/"
```

**To improve scraping**, the system:
- Uses proper User-Agent headers
- Follows same-domain links
- Extracts content from semantic HTML elements (`<main>`, `<article>`, etc.)
- Cleans whitespace and formatting

---

## Rebuilding the Index

After adding or updating content, rebuild the vector database:

### Step 1: Ensure Dependencies
```bash
pip install -r requirements_hf.txt
```

### Step 2: Set Environment Variables
Make sure your `.env` file contains:
```env
COHEREAPIKEY=your_cohere_api_key
OPENROUTER_API_KEY=your_openrouter_api_key  # Optional but recommended
```

### Step 3: Run the Build Script
```bash
python build_index.py
```

**What Happens**:
1. ✅ Scrapes portfolio website
2. ✅ Loads all PDFs from `docs/pdfs/`
3. ✅ Loads all transcripts from `docs/youtube/`
4. ✅ Loads all READMEs from `docs/readmes/`
5. ✅ Splits documents into chunks (1000 chars, 200 overlap)
6. ✅ Creates embeddings using Cohere API
7. ✅ Builds FAISS vector index
8. ✅ Saves index files to `docs/faiss/`

**Expected Output**:
```
=== Building FAISS Index for Julien Serbanescu RAG System ===

1. Scraping portfolio website...
   Scraped 12345 characters from portfolio website

2. Loading PDF documents...
   Processing PDF: resume.pdf
   Processing PDF: paper.pdf
   Loaded 2 PDF pages

3. Loading audio transcripts...
   Processing transcript: interview_transcript.txt
   Loaded 1 audio transcripts

4. Loading README documents...
   Processing README: Project1-README.md
   Processing README: Project2-README.md
   Loaded 15 README documents

Total documents collected: 19

5. Splitting documents into chunks...
   Created 45 document chunks

6. Creating embeddings...
   Created embedding matrix with shape: (45, 4096)

7. Building FAISS index...
   FAISS index built with 45 vectors

8. Creating docstore and mapping...
9. Saving index files...
   Saved FAISS index to: docs/faiss/index.faiss
   Saved docstore to: docs/faiss/index.pkl
   Saved metadata to: docs/faiss/metadata.pkl

10. Creating document lookup file...
   Saved document lookup to: docs/faiss/document_lookup.txt

=== Index building completed successfully! ===
Total documents indexed: 45
Index files saved to: docs/faiss/
```

### Step 4: Test the System
```bash
python queryrun.py
```

---

## Complete Workflow Example

Here's a complete example of updating your content:

```bash
# 1. Clear old database
rm docs/faiss/*.faiss docs/faiss/*.pkl docs/faiss/*.txt

# 2. Add new PDFs
cp ~/Documents/new_resume.pdf docs/pdfs/

# 3. Download new repos
python download_readmes.py --username DaJulster --token YOUR_TOKEN

# 4. Add manual READMEs
cp ~/Projects/MyNewProject/README.md docs/readmes/MyNewProject-README.md

# 5. Rebuild index
python build_index.py

# 6. Test
python queryrun.py
```

---

## Troubleshooting

### Issue: "No documents found"
**Solution**: Check that files exist in the expected directories:
- `docs/pdfs/*.pdf`
- `docs/youtube/*_transcript.txt`
- `docs/readmes/*.md`

### Issue: "COHEREAPIKEY not found"
**Solution**: Add your Cohere API key to `.env`:
```env
COHEREAPIKEY=your_key_here
```

### Issue: Portfolio scraping fails
**Solution**: 
- Check internet connection
- Verify the URL is accessible
- Check if the website blocks scrapers (may need to adjust User-Agent)

### Issue: README download rate limits
**Solution**: 
- Use a GitHub Personal Access Token
- Wait between runs
- Use `--exclude-forks` to reduce number of repos

### Issue: Index files not updating
**Solution**: 
- Make sure you deleted old index files
- Check file permissions
- Verify the `docs/faiss/` directory exists

---

## Best Practices

1. **Regular Updates**: Rebuild the index monthly or when adding significant new content
2. **Version Control**: Don't commit index files (they're in `.gitignore`)
3. **Backup**: Keep backups of important source documents
4. **Testing**: Always test queries after rebuilding to ensure quality
5. **Documentation**: Keep README files well-formatted for better retrieval

---

## Next Steps

After rebuilding:
- Test queries with `python queryrun.py`
- Deploy updates if using the web API
- Monitor query quality and adjust chunk sizes if needed

For questions or issues, check the main README.md or open an issue on GitHub.