Spaces:

DaJulster
/

julienserbanescu-rag

Sleeping

import os
import shutil

faiss_dir = "docs/faiss"
if os.path.exists(faiss_dir):
    for file in ["index.faiss", "index.pkl", "metadata.pkl", "document_lookup.txt"]:
        filepath = os.path.join(faiss_dir, file)
        if os.path.exists(filepath):
            os.remove(filepath)
            print(f"Deleted {file}")

Adding New Content

1. Adding PDF Documents

Location: docs/pdfs/

Steps:

Copy your PDF files to docs/pdfs/
Supported formats: .pdf
The system automatically extracts text from all PDFs in this directory
Each page is processed separately for better retrieval

Example:

# Copy new PDFs
cp /path/to/your/resume.pdf docs/pdfs/
cp /path/to/your/paper.pdf docs/pdfs/

2. Adding Audio Transcripts

Location: docs/youtube/

Steps:

Place transcript files in docs/youtube/
Naming convention: *_transcript.txt
The system automatically processes all files matching this pattern

Example:

# Add transcript
cp interview_transcript.txt docs/youtube/MyInterview_transcript.txt

3. Adding README Files Manually

Location: docs/readmes/

Steps:

Copy README files to docs/readmes/
Naming convention: *-README.md (recommended)
Files containing "Tpoze-subnet" (case-insensitive) are automatically excluded

Example:

# Add a README manually
cp /path/to/project/README.md docs/readmes/MyProject-README.md

Adding New Repositories

Method 1: Using the Download Script (Recommended)

The download_readmes.py script automatically downloads READMEs from your GitHub repositories.

Steps:

Get a GitHub Personal Access Token (optional but recommended):
- Go to GitHub Settings → Developer settings → Personal access tokens
- Create a token with public_repo scope
- Copy the token
Run the download script:

# Basic usage (uses public API, rate-limited)
python download_readmes.py --username DaJulster

# With token (higher rate limits)
python download_readmes.py --username DaJulster --token YOUR_GITHUB_TOKEN

# Show available repos without downloading
python download_readmes.py --username DaJulster --show-available

# Filter options
python download_readmes.py --username DaJulster \
    --exclude-forks \
    --min-stars 1 \
    --exclude-tpoze \
    --contribution-type owner

Script Options:

--username: Your GitHub username (required)
--token: GitHub personal access token (optional, increases rate limit)
--output-dir: Output directory (default: docs/readmes)
--exclude-forks: Exclude forked repositories
--min-stars: Minimum stars required
--exclude-tpoze: Exclude repos containing 'tpoze' in name
--contribution-type: Filter by owner, contributor, committer, or all
--show-available: Show repos without downloading

What the Script Does:

Finds all repositories you own
Finds repositories you've contributed to (commits, PRs, issues)
Checks organizations you're a member of
Downloads README files from each repository
Saves them with metadata to docs/readmes/

Method 2: Manual Addition

Clone or download the repository
Copy the README file:

cp /path/to/repo/README.md docs/readmes/RepoName-README.md

Optionally add metadata at the top of the file:

# Repository Name
**Repository:** owner/repo-name
**Contribution:** 👑 Owner
**Stars:** 42
**Description:** Project description
**Language:** Python
**Last Updated:** 2024-01-01
---

Adding Portfolio Website Data

The portfolio website is automatically scraped during indexing. The system:

Scrapes the main page: https://julien-ser.github.io/JulienSerbanescu/
Follows internal links: Automatically discovers and scrapes linked pages (up to 5 pages)
Extracts main content: Focuses on main content areas, ignoring navigation/footers
Handles dynamic content: Removes scripts and styles for clean text extraction

To change the portfolio URL, edit build_index.py:

portfolio_url = "https://your-portfolio-url.com/"

To improve scraping, the system:

Uses proper User-Agent headers
Follows same-domain links
Extracts content from semantic HTML elements (<main>, <article>, etc.)
Cleans whitespace and formatting

Rebuilding the Index

After adding or updating content, rebuild the vector database:

Step 1: Ensure Dependencies

pip install -r requirements_hf.txt

Step 2: Set Environment Variables

Make sure your .env file contains:

COHEREAPIKEY=your_cohere_api_key
OPENROUTER_API_KEY=your_openrouter_api_key  # Optional but recommended

Step 3: Run the Build Script

python build_index.py

What Happens:

✅ Scrapes portfolio website
✅ Loads all PDFs from docs/pdfs/
✅ Loads all transcripts from docs/youtube/
✅ Loads all READMEs from docs/readmes/
✅ Splits documents into chunks (1000 chars, 200 overlap)
✅ Creates embeddings using Cohere API
✅ Builds FAISS vector index
✅ Saves index files to docs/faiss/

Expected Output:

=== Building FAISS Index for Julien Serbanescu RAG System ===

1. Scraping portfolio website...
   Scraped 12345 characters from portfolio website

2. Loading PDF documents...
   Processing PDF: resume.pdf
   Processing PDF: paper.pdf
   Loaded 2 PDF pages

3. Loading audio transcripts...
   Processing transcript: interview_transcript.txt
   Loaded 1 audio transcripts

4. Loading README documents...
   Processing README: Project1-README.md
   Processing README: Project2-README.md
   Loaded 15 README documents

Total documents collected: 19

5. Splitting documents into chunks...
   Created 45 document chunks

6. Creating embeddings...
   Created embedding matrix with shape: (45, 4096)

7. Building FAISS index...
   FAISS index built with 45 vectors

8. Creating docstore and mapping...
9. Saving index files...
   Saved FAISS index to: docs/faiss/index.faiss
   Saved docstore to: docs/faiss/index.pkl
   Saved metadata to: docs/faiss/metadata.pkl

10. Creating document lookup file...
   Saved document lookup to: docs/faiss/document_lookup.txt

=== Index building completed successfully! ===
Total documents indexed: 45
Index files saved to: docs/faiss/

Step 4: Test the System

python queryrun.py

Complete Workflow Example

Here's a complete example of updating your content:

# 1. Clear old database
rm docs/faiss/*.faiss docs/faiss/*.pkl docs/faiss/*.txt

# 2. Add new PDFs
cp ~/Documents/new_resume.pdf docs/pdfs/

# 3. Download new repos
python download_readmes.py --username DaJulster --token YOUR_TOKEN

# 4. Add manual READMEs
cp ~/Projects/MyNewProject/README.md docs/readmes/MyNewProject-README.md

# 5. Rebuild index
python build_index.py

# 6. Test
python queryrun.py

Troubleshooting

Issue: "No documents found"

Solution: Check that files exist in the expected directories:

docs/pdfs/*.pdf
docs/youtube/*_transcript.txt
docs/readmes/*.md

Issue: "COHEREAPIKEY not found"

Solution: Add your Cohere API key to .env:

COHEREAPIKEY=your_key_here

Issue: Portfolio scraping fails

Solution:

Check internet connection
Verify the URL is accessible
Check if the website blocks scrapers (may need to adjust User-Agent)

Issue: README download rate limits

Solution:

Use a GitHub Personal Access Token
Wait between runs
Use --exclude-forks to reduce number of repos

Issue: Index files not updating

Solution:

Make sure you deleted old index files
Check file permissions
Verify the docs/faiss/ directory exists

Best Practices

Regular Updates: Rebuild the index monthly or when adding significant new content
Version Control: Don't commit index files (they're in .gitignore)
Backup: Keep backups of important source documents
Testing: Always test queries after rebuilding to ensure quality
Documentation: Keep README files well-formatted for better retrieval

Next Steps

After rebuilding:

Test queries with python queryrun.py
Deploy updates if using the web API
Monitor query quality and adjust chunk sizes if needed

For questions or issues, check the main README.md or open an issue on GitHub.

Guide: Updating Your RAG System Content

Table of Contents

Clearing the Vector Database

Method 1: Manual Deletion

Method 2: Using Python

Adding New Content

1. Adding PDF Documents

2. Adding Audio Transcripts

3. Adding README Files Manually

Adding New Repositories

Method 1: Using the Download Script (Recommended)

Method 2: Manual Addition

Adding Portfolio Website Data

Rebuilding the Index

Step 1: Ensure Dependencies

Step 2: Set Environment Variables

Step 3: Run the Build Script

Step 4: Test the System

Complete Workflow Example

Troubleshooting

Issue: "No documents found"

Issue: "COHEREAPIKEY not found"

Issue: Portfolio scraping fails

Issue: README download rate limits

Issue: Index files not updating

Best Practices

Next Steps