Index all 5 publications; add self-updating refresh pipeline

The knowledge base held 1 of 5 papers. download_papers.py only queried arXiv,
so anything published solely to ACM DL or PMLR was invisible to the RAG,
including both Canadian AI 2026 papers.

download_publications.py (replaces download_papers.py)
Merges four sources on a normalised title:
openalex widest coverage, picks up new work on its own, but lags months
crossref authoritative published author order for DOI-bearing papers
pmlr scrapes configured volume indexes; PMLR reaches OpenAlex late
arxiv preprints
Plus a manual list in sources.json for anything all four miss.

Author matching requires family name AND a known given name on the same
entry. Crossref returns several unrelated Serbanescus, and a surname-only
match pulled them all in.

Crossref order is treated as canonical: it corrected two papers where the
author order was previously wrong.

refresh_index.py
Rewritten around the failure mode that mattered. It used to run every step,
ignore failures, and rebuild regardless, so a rate-limited GitHub run
silently rebuilt the index from a half-empty docs/ tree.

Now aborts before rebuilding when a source fails, or when the corpus shrinks
more than 25% against the last run (an API returning [] rather than an
error). On abort nothing is rebuilt and nothing is pushed, so the deployed
index keeps serving the last good build. State in .refresh_state.json.

sync_resume.py
Copies the current resume into docs/pdfs so answers track the live document.
Removed the stale April copy, which would have contradicted it.

.gitignore
Stopped ignoring docs/faiss/ and docs/pdfs/. The Space serves both directly
out of the repo; the rules only appeared to work because those files were
committed before the rules existed, which meant any newly added PDF never
reached the deployment.

Scheduling
scheduled_refresh.sh for cron (flock, venv, rotating logs) and a GitHub
Actions workflow. AUTOMATION.md covers both, what updates automatically,
and why LinkedIn cannot.

All 5 papers now retrieve as top hits against the rebuilt index.

.github/workflows/refresh-knowledge-base.yml

@@ -0,0 +1,84 @@
+# Requires this repo to be mirrored to GitHub. See AUTOMATION.md for setup, or
+# use the julienlab cron option instead if you would rather not mirror.
+
+name: Refresh RAG knowledge base
+
+on:
+  schedule:
+    # 07:00 UTC every Monday (03:00 Toronto in EDT, 02:00 in EST).
+    - cron: "0 7 * * 1"
+  workflow_dispatch:
+    inputs:
+      allow_partial:
+        description: "Rebuild even if a source fails"
+        type: boolean
+        default: false
+
+concurrency:
+  group: refresh-knowledge-base
+  cancel-in-progress: false
+
+jobs:
+  refresh:
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements_hf.txt
+
+      - name: Refresh knowledge base and rebuild index
+        env:
+          COHEREAPIKEY: ${{ secrets.COHEREAPIKEY }}
+          COHERE_API_KEY: ${{ secrets.COHEREAPIKEY }}
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+          # Lets the GitHub sources use the 5000 req/hr authenticated limit
+          # instead of the 60 req/hr anonymous one.
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          python refresh_index.py ${{ inputs.allow_partial && '--allow-partial' || '' }}
+
+      - name: Push updated index to Hugging Face Space
+        if: success()
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${HF_TOKEN:-}" ]; then
+            echo "HF_TOKEN not set; skipping deploy."
+            exit 0
+          fi
+
+          git config user.name  "rag-refresh-bot"
+          git config user.email "actions@github.com"
+          git add docs/ .refresh_state.json
+
+          if git diff --staged --quiet; then
+            echo "Index unchanged; nothing to deploy."
+            exit 0
+          fi
+
+          git commit -m "chore: refresh knowledge base ($(date -u +%Y-%m-%d))"
+          git push "https://oauth:${HF_TOKEN}@huggingface.co/spaces/DaJulster/julienserbanescu-rag" HEAD:main
+          echo "Deployed. The Space will rebuild automatically."
+
+      - name: Upload run log
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: refresh-state
+          path: .refresh_state.json
+          if-no-files-found: ignore

.gitignore

@@ -1,9 +1,19 @@
-docs/faiss/index.faiss
-docs/pdfs/*
-docs/youtube/*
-.git_backup
-__pycache__
-*.pyc
+# Secrets
 .env
 pat.txt
-/venv
+
+# Local environment
+/venv
+__pycache__
+*.pyc
+.git_backup
+
+# Transcripts are large and not currently part of the pipeline
+docs/youtube/*
+
+# NOTE: docs/faiss/ and docs/pdfs/ are deliberately NOT ignored.
+# The Hugging Face Space serves the prebuilt index and source PDFs directly out
+# of this repo, so ignoring them silently strips content from the deployment.
+# They were previously ignored and only kept working because the files had been
+# committed before the rule was added, which meant any newly added PDF (such as
+# a refreshed resume) never reached the Space.

.refresh_state.json

@@ -0,0 +1,9 @@
+{
+  "corpus": {
+    "publications": 5,
+    "readmes": 70,
+    "github_activity": 18,
+    "pdfs": 4
+  },
+  "last_run": "2026-08-08T00:30:21.863716+00:00"
+}

AUTOMATION.md

@@ -0,0 +1,174 @@
+# Keeping the RAG current
+
+The knowledge base is a set of files under `docs/` plus a prebuilt FAISS index.
+The Hugging Face Space serves that index directly out of this repo, so "updating
+the RAG" means: re-download sources, rebuild the index, commit, push.
+
+`refresh_index.py` does all four steps.
+
+```bash
+python refresh_index.py              # download + rebuild, no deploy
+python refresh_index.py --push       # download + rebuild + deploy to the Space
+python refresh_index.py --dry-run    # print the plan, touch nothing
+```
+
+---
+
+## What updates automatically, and what does not
+
+| Source | Path | Automatic? | How |
+|---|---|---|---|
+| Publications | `docs/papers/` | **Yes** | `download_publications.py` merges OpenAlex, Crossref, PMLR, arXiv |
+| GitHub repo READMEs | `docs/readmes/` | **Yes** | `download_readmes.py` |
+| GitHub issues + PRs | `docs/github_activity/` | **Yes** | `download_github_activity.py` |
+| Resume | `docs/pdfs/` | **Yes** | `sync_resume.py` copies from the path in `sources.json` |
+| Portfolio site text | scraped at build | **Yes** | `build_index.py` scrapes the live site |
+| LinkedIn | `docs/linkedin/` | **No** | Manual export, see below |
+
+### Publications
+
+`download_publications.py` queries four sources and merges them on a normalised
+title, so one paper appearing in three places becomes one document.
+
+Each source exists because the others have a gap:
+
+- **OpenAlex** is the widest net and picks up new work on its own, but lags
+  months behind publication.
+- **Crossref** has authoritative author order for anything with a DOI. This
+  matters: the portfolio previously listed the wrong first author on two papers.
+- **PMLR** proceedings (Canadian AI, ICML, AISTATS) are not in OpenAlex for a
+  long time, so configured volumes are scraped directly.
+- **arXiv** catches preprints.
+
+**When you publish somewhere new**, one of two things happens:
+
+- Venue has a DOI or is on arXiv: nothing to do, it appears within a few weeks.
+- Venue is a new PMLR volume: add the volume to `sources.json`:
+
+  ```json
+  "pmlr": { "volumes": ["v318", "v3XX"] }
+  ```
+
+- Neither: add it to `manual_publications` in `sources.json`. Entries there are
+  always included.
+
+Verify with `python download_publications.py --dry-run`, which lists everything
+found and flags anything missing an abstract.
+
+### LinkedIn
+
+There is no API for reading your own profile. LinkedIn removed public profile
+API access, and scraping it violates their terms and will get the account
+flagged. The supported path is the periodic data export:
+
+1. <https://www.linkedin.com/mypreferences/d/download-my-data>
+2. Request the archive, wait for the email (usually under 24 hours)
+3. Unzip into `docs/linkedin/`
+4. `python refresh_index.py --skip-download` to rebuild
+
+`build_index.py` already reads `Profile.csv`, `Positions.csv`, `Education.csv`,
+`Skills.csv`, `Projects.csv`, and any `.txt` / `.md` in that directory.
+
+Realistically the resume PDF already covers most of what LinkedIn holds, and it
+updates automatically. Treat the export as an occasional top-up, not a routine.
+
+---
+
+## Safety guards
+
+A scheduled job that quietly degrades the index is worse than no job. The
+refresh aborts rather than rebuilding when:
+
+- **any source fails** (`--allow-partial` overrides), so a rate-limited GitHub
+  run cannot bake missing repos into the index;
+- **the corpus shrinks more than 25%** against the last recorded run
+  (`--max-shrink N` to adjust), which catches an API returning an empty list
+  instead of an error.
+
+On abort, nothing is rebuilt and nothing is pushed. The deployed index keeps
+serving the last good build. State lives in `.refresh_state.json`.
+
+---
+
+## Scheduling
+
+Two options. Pick one.
+
+### Option A: cron on julienlab (recommended)
+
+Best fit given the box is already on and already holds backups. No secrets leave
+the network.
+
+```bash
+# on julienlab, one time
+git clone https://huggingface.co/spaces/DaJulster/julienserbanescu-rag
+cd julienserbanescu-rag
+python3 -m venv venv
+./venv/bin/pip install -r requirements_hf.txt
+
+# secrets
+cat > .env <<'EOF'
+COHEREAPIKEY=...
+COHERE_API_KEY=...
+OPENROUTER_API_KEY=...
+GITHUB_TOKEN=...
+EOF
+chmod 600 .env
+
+# let git push to HF without prompting
+git remote set-url origin https://oauth:<HF_TOKEN>@huggingface.co/spaces/DaJulster/julienserbanescu-rag
+
+# point sources.json at wherever the resume lives on this box,
+# or drop the resume path entirely and let the PDF already in docs/pdfs stand
+
+crontab -e
+```
+
+Add:
+
+```
+0 3 * * 1  /home/julien/julienserbanescu-rag/scheduled_refresh.sh
+```
+
+`scheduled_refresh.sh` handles the working directory, the venv, a `flock` so
+overlapping runs cannot corrupt the index, and rotating logs in `logs/`.
+
+Check a run with `tail -n 60 logs/refresh-*.log | less`.
+
+### Option B: GitHub Actions
+
+Use this if you would rather not depend on julienlab being up. It needs the repo
+mirrored to GitHub, because Hugging Face does not run Actions.
+
+```bash
+git remote add github https://github.com/Julien-ser/julienserbanescu-rag.git
+git push github main
+```
+
+Then add repo secrets under Settings → Secrets and variables → Actions:
+
+| Secret | Purpose |
+|---|---|
+| `COHEREAPIKEY` | Embeddings for the index build |
+| `OPENROUTER_API_KEY` | Response generation |
+| `HF_TOKEN` | Write token so the workflow can push to the Space |
+
+`.github/workflows/refresh-knowledge-base.yml` then runs every Monday 07:00 UTC,
+and on demand via workflow_dispatch. `GITHUB_TOKEN` is provided by Actions
+automatically, which also lifts the GitHub API limit from 60 to 5000 req/hour.
+
+Note that this pushes from GitHub to Hugging Face, so GitHub becomes the source
+of truth. Do not also commit directly to the HF remote or the two will diverge.
+
+---
+
+## Tokens
+
+`pat.txt` holds an expired GitHub PAT (it returns 401). Nothing reads it by
+preference; the scripts take `--token` or `GITHUB_TOKEN`. Put a fresh
+fine-grained PAT with **public repo read** scope in `.env` as `GITHUB_TOKEN` and
+delete `pat.txt`.
+
+Without a working token the GitHub sources fall back to 60 requests/hour, which
+is not enough for 70 repos, so those steps will fail and the refresh will
+correctly abort instead of shrinking the index.

build_index.py

@@ -24,7 +24,7 @@ import cohere
 from dotenv import load_dotenv
 from langchain_community.docstore.document import Document
 from langchain_community.docstore.in_memory import InMemoryDocstore
-from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain_text_splitters import RecursiveCharacterTextSplitter
 import requests
 from bs4 import BeautifulSoup
 import traceback
@@ -357,6 +357,112 @@ def load_readme_documents(readme_dir):
     
     return documents
 
+def load_github_activity(activity_dir):
+    """Load GitHub issues and PRs downloaded by download_github_activity.py"""
+    documents = []
+    if not os.path.exists(activity_dir):
+        print(f"  GitHub activity directory {activity_dir} not found, skipping...")
+        return documents
+
+    md_files = [f for f in os.listdir(activity_dir) if f.endswith('.md')]
+    for md_file in md_files:
+        try:
+            filepath = os.path.join(activity_dir, md_file)
+            with open(filepath, 'r', encoding='utf-8') as f:
+                content = f.read()
+
+            doc_type = 'github_pr' if '_PR_' in md_file else 'github_issue'
+            doc = Document(
+                page_content=content,
+                metadata={
+                    'source': f"docs/github_activity/{md_file}",
+                    'type': doc_type,
+                }
+            )
+            documents.append(doc)
+        except Exception as e:
+            print(f"  Error loading {md_file}: {e}")
+    return documents
+
+
+def load_publications(papers_dir):
+    """Load publication documents written by download_publications.py.
+
+    Covers every source that script merges (OpenAlex, Crossref, PMLR, arXiv,
+    manual), not just arXiv.
+    """
+    documents = []
+    if not os.path.exists(papers_dir):
+        print(f"  Papers directory {papers_dir} not found, skipping...")
+        return documents
+
+    md_files = [f for f in os.listdir(papers_dir) if f.endswith('.md')]
+    for md_file in md_files:
+        try:
+            filepath = os.path.join(papers_dir, md_file)
+            with open(filepath, 'r', encoding='utf-8') as f:
+                content = f.read()
+
+            doc = Document(
+                page_content=content,
+                metadata={
+                    'source': f"docs/papers/{md_file}",
+                    'type': 'publication',
+                }
+            )
+            documents.append(doc)
+        except Exception as e:
+            print(f"  Error loading {md_file}: {e}")
+    return documents
+
+
+def load_linkedin_export(linkedin_dir):
+    """Load LinkedIn data export files.
+    
+    LinkedIn lets you download your data at: https://www.linkedin.com/mypreferences/d/download-my-data
+    Place the exported CSV/JSON files in docs/linkedin/.
+    Supports: Profile.csv, Positions.csv, Education.csv, Skills.csv, Projects.csv, and any .txt/.md files.
+    """
+    documents = []
+    if not os.path.exists(linkedin_dir):
+        print(f"  LinkedIn directory {linkedin_dir} not found, skipping...")
+        print(f"  To add LinkedIn data: export from https://www.linkedin.com/mypreferences/d/download-my-data")
+        print(f"  Then place files in {linkedin_dir}/")
+        return documents
+
+    import csv
+    for filename in os.listdir(linkedin_dir):
+        filepath = os.path.join(linkedin_dir, filename)
+        try:
+            if filename.endswith('.csv'):
+                with open(filepath, 'r', encoding='utf-8', errors='replace') as f:
+                    reader = csv.DictReader(f)
+                    rows = list(reader)
+                if not rows:
+                    continue
+                content = f"# LinkedIn Data: {filename}\n\n"
+                for row in rows:
+                    content += "\n".join(f"**{k}:** {v}" for k, v in row.items() if v) + "\n---\n"
+                doc = Document(
+                    page_content=content,
+                    metadata={'source': f"docs/linkedin/{filename}", 'type': 'linkedin_export'}
+                )
+                documents.append(doc)
+            elif filename.endswith(('.txt', '.md')):
+                with open(filepath, 'r', encoding='utf-8', errors='replace') as f:
+                    content = f.read()
+                if content.strip():
+                    doc = Document(
+                        page_content=content,
+                        metadata={'source': f"docs/linkedin/{filename}", 'type': 'linkedin_export'}
+                    )
+                    documents.append(doc)
+        except Exception as e:
+            print(f"  Error loading {filename}: {e}")
+
+    return documents
+
+
 def main():
     """Main function to build the FAISS index"""
     print("=== Building FAISS Index for Julien Serbanescu RAG System ===")
@@ -389,25 +495,11 @@ def main():
     else:
         print("   Warning: Could not scrape portfolio website content")
     
-    # 1b. Scrape LinkedIn profile (optional - may require authentication)
-    print("\n1b. Scraping LinkedIn profile (if URL provided)...")
-    linkedin_url = os.getenv("LINKEDIN_PROFILE_URL")  # Optional: set in .env
-    if linkedin_url:
-        linkedin_content = scrape_linkedin_profile(linkedin_url)
-        if linkedin_content:
-            doc = Document(
-                page_content=linkedin_content,
-                metadata={
-                    'source': linkedin_url,
-                    'type': 'linkedin_profile'
-                }
-            )
-            all_documents.append(doc)
-            print(f"   Scraped {len(linkedin_content)} characters from LinkedIn profile")
-        else:
-            print("   Warning: Could not scrape LinkedIn profile (may require authentication)")
-    else:
-        print("   Skipping LinkedIn (set LINKEDIN_PROFILE_URL in .env to enable)")
+    # 1b. Load LinkedIn export data
+    print("\n1b. Loading LinkedIn export data...")
+    linkedin_docs = load_linkedin_export("docs/linkedin")
+    all_documents.extend(linkedin_docs)
+    print(f"   Loaded {len(linkedin_docs)} LinkedIn documents")
     
     # 2. Load PDF documents
     print("\n2. Loading PDF documents...")
@@ -429,6 +521,18 @@ def main():
     readme_docs = load_readme_documents(readme_dir)
     all_documents.extend(readme_docs)
     print(f"   Loaded {len(readme_docs)} README documents")
+
+    # 5. Load GitHub issues & PRs
+    print("\n5. Loading GitHub issues & PRs...")
+    activity_docs = load_github_activity("docs/github_activity")
+    all_documents.extend(activity_docs)
+    print(f"   Loaded {len(activity_docs)} GitHub issues/PRs")
+
+    # 6. Load publications (all sources, not just arXiv)
+    print("\n6. Loading publications...")
+    paper_docs = load_publications("docs/papers")
+    all_documents.extend(paper_docs)
+    print(f"   Loaded {len(paper_docs)} publications")
     
     print(f"\nTotal documents collected: {len(all_documents)}")
     
@@ -436,8 +540,8 @@ def main():
         print("No documents found! Please check your document directories.")
         return
     
-    # 5. Split documents into chunks
-    print("\n5. Splitting documents into chunks...")
+    # 7. Split documents into chunks
+    print("\n7. Splitting documents into chunks...")
     text_splitter = RecursiveCharacterTextSplitter(
         chunk_size=1000,
         chunk_overlap=200,
@@ -447,8 +551,8 @@ def main():
     split_documents = text_splitter.split_documents(all_documents)
     print(f"   Created {len(split_documents)} document chunks")
     
-    # 6. Create embeddings
-    print("\n6. Creating embeddings...")
+    # 8. Create embeddings
+    print("\n8. Creating embeddings...")
     texts = [doc.page_content for doc in split_documents]
     embeddings = embedding_function.embed_documents(texts)
     
@@ -456,8 +560,8 @@ def main():
     embedding_matrix = np.array(embeddings).astype('float32')
     print(f"   Created embedding matrix with shape: {embedding_matrix.shape}")
     
-    # 7. Build FAISS index
-    print("\n7. Building FAISS index...")
+    # 9. Build FAISS index
+    print("\n9. Building FAISS index...")
     dimension = embedding_matrix.shape[1]
     index = faiss.IndexFlatIP(dimension)  # Inner product for cosine similarity
     
@@ -467,8 +571,8 @@ def main():
     
     print(f"   FAISS index built with {index.ntotal} vectors")
     
-    # 8. Create docstore and mapping
-    print("\n8. Creating docstore and mapping...")
+    # 10. Create docstore and mapping
+    print("\n10. Creating docstore and mapping...")
     docstore = InMemoryDocstore()
     index_to_docstore_id = {}
     
@@ -477,8 +581,8 @@ def main():
         docstore.add({doc_id: doc})
         index_to_docstore_id[i] = doc_id
     
-    # 9. Save everything
-    print("\n9. Saving index files...")
+    # 11. Save everything
+    print("\n11. Saving index files...")
     
     # Save FAISS index
     faiss_index_path = os.path.join(output_dir, "index.faiss")
@@ -498,8 +602,8 @@ def main():
         pickle.dump(metadata_list, f)
     print(f"   Saved metadata to: {metadata_path}")
     
-    # 10. Create human-readable document lookup
-    print("\n10. Creating document lookup file...")
+    # 12. Create human-readable document lookup
+    print("\n12. Creating document lookup file...")
     lookup_path = os.path.join(output_dir, "document_lookup.txt")
     with open(lookup_path, 'w', encoding='utf-8') as f:
         for i, doc in enumerate(split_documents):

docs/faiss/index.faiss

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:761c3e1c863aed9b8194abb693fb5cd07c237ebf4987efd4f58ab4349df6cbd7
-size 3424301
+oid sha256:c939addb445c9cb242b77a382f005eb0bcaf042dd36ea251e20752ea421ef36d
+size 3477549

docs/faiss/index.pkl

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8a2bb4a67ed657a46ea93c309f4f686b6c7115bd2b18c7605c5b28c13d686ff1
-size 767874
+oid sha256:d0a6ff76940741bcfc061f6e2a926603fe29b7c680ad371a6f90d91535aa6eb9
+size 776917

docs/faiss/metadata.pkl

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:16da6051eb9ab58343238c7c7dd50c32229abdfab355d06871f7d817904ef93d
-size 16460
+oid sha256:e37da3ecfb523e1d4f7e6f5b247a927e6fa671e4fe14b1455bf1d5f251b58080
+size 17099

docs/papers/2024_UnAnswGen_A_Systematic_Approach_for_Generating_Unanswerable_Questions_in_Machine_Reading_Comprehension.md

@@ -0,0 +1,14 @@
+# UnAnswGen: A Systematic Approach for Generating Unanswerable Questions in Machine Reading Comprehension
+
+**Authors:** Hadiseh Moradisani, Fattane Zarrinkalam, Julien Serbanescu, Zeinab Noorian
+**Venue:** Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region
+**Year:** 2024
+**DOI:** 10.1145/3673791.3698413
+**URL:** https://doi.org/10.1145/3673791.3698413
+**Indexed from:** openalex, crossref
+
+---
+
+## Abstract
+
+This paper introduces a configurable software workflow to automatically generate and publicly share a dataset of multi-labeled unanswerable questions for Machine Reading Comprehension (MRC). Unlike existing datasets like SQuAD2.0, which do not account for the reasons behind question unanswerability, our method fills a critical gap by systematically transforming answerable questions into their unanswerable counterparts across various linguistic dimensions including entity swap, number swap, negation, antonym, mutual exclusion, and no information. These candidate unanswerable questions are evaluated using advanced MRC models to ensure their context-based unanswerability, with the final selection based on a majority consensus mechanism. Our approach addresses the scarcity of multi-labeled datasets like SQuAD2-CR, enabling comprehensive evaluation of MRC systems' ability to handle unanswerable queries and facilitating the exploration of solutions such as query reformulation. The resulting UnAnswGen dataset and associated software workflow are made publicly available to advance research in machine reading comprehension, offering researchers a standardized toolset for evaluating and enhancing MRC systems' robustness and performance.

docs/papers/2025_FalseCoTQA_Adversarial_Multi-Hop_QA_via_Knowledge-Grounded_False_Chains_of_Thought.md

@@ -0,0 +1,14 @@
+# FalseCoTQA: Adversarial Multi-Hop QA via Knowledge-Grounded False Chains of Thought
+
+**Authors:** Julien Serbanescu, Mahdiyar Ali Akbar Alavi, Faezeh Ensan, Fattane Zarrinkalam
+**Venue:** Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region
+**Year:** 2025
+**DOI:** 10.1145/3767695.3769494
+**URL:** https://doi.org/10.1145/3767695.3769494
+**Indexed from:** openalex, crossref
+
+---
+
+## Abstract
+
+Multi-hop question answering (QA) models excel at decomposing complex queries into sequential reasoning steps, yet they remain vulnerable to subtly flawed inference chains that appear reasonable but are factually incorrect. To quantify and address this weakness, we present FalseCoTQA, an adversarial benchmark that injects knowledge-grounded false reasoning into retrieval-augmented contexts. Unlike prior methods that merely tweak surface text, FalseCoTQA leverages a domain-agnostic knowledge graph to systematically replace entities to construct semantically coherent yet incorrect chains of thought on top of standard multi-hop datasets (HotpotQA and MuSiQue). By evaluating state-of-the-art language models on this benchmark, we observe dramatic drops in answer accuracy, highlighting their tendency to follow deceptive reasoning without verifying factual consistency. We expect the proposed benchmark to contribute to the evaluation and improvement of the robustness and reliability of language models in multi-hop question answering.

docs/papers/2025_Uncovering_the_Persuasive_Fingerprint_of_LLMs_in_Jailbreaking_Attacks.md

@@ -0,0 +1,16 @@
+# Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks
+
+**Authors:** Havva Alizadeh Noughabi, Julien Serbanescu, Fattane Zarrinkalam, Ali Dehghantanha
+**Venue:** Proceedings of the 34th ACM International Conference on Information and Knowledge Management
+**Year:** 2025
+**DOI:** 10.1145/3746252.3760929
+**arXiv ID:** 2510.21983v1
+**URL:** https://doi.org/10.1145/3746252.3760929
+**PDF:** https://arxiv.org/pdf/2510.21983v1
+**Indexed from:** openalex, crossref, arxiv
+
+---
+
+## Abstract
+
+Despite recent advances, Large Language Models (LLMs) remain vulnerable to jailbreak attacks that bypass alignment safeguards and elicit harmful outputs. While prior research has proposed various attack strategies differing in human readability and transferability, little attention has been paid to the linguistic and psychological mechanisms that may influence a model's susceptibility to such attacks. In this paper, we examine an interdisciplinary line of research that leverages foundational theories of persuasion from the social sciences to craft adversarial prompts capable of circumventing alignment constraints in LLMs. Drawing on well-established persuasive strategies, we hypothesize that LLMs, having been trained on large-scale human-generated text, may respond more compliantly to prompts with persuasive structures. Furthermore, we investigate whether LLMs themselves exhibit distinct persuasive fingerprints that emerge in their jailbreak responses. Empirical evaluations across multiple aligned LLMs reveal that persuasion-aware prompts significantly bypass safeguards, demonstrating their potential to induce jailbreak behaviors. This work underscores the importance of cross-disciplinary insight in addressing the evolving challenges of LLM safety. The code and data are available. https://github.com/CyberScienceLab/Our-Papers/tree/main/PersuasiveJailbreaking/.

docs/papers/2026_Cause-Conditioned_Multi-Task_Learning_for_Answerable_Question_Suggestion_in_MRC.md

@@ -0,0 +1,13 @@
+# Cause-Conditioned Multi-Task Learning for Answerable Question Suggestion in MRC
+
+**Authors:** Hadiseh Moradisani, Fattane Zarrinkalam, Julien Serbanescu, Zeinab Noorian
+**Venue:** Proceedings of the The 39th Canadian Conference on Artificial Intelligence
+**Year:** 2026
+**URL:** https://proceedings.mlr.press/v318/moradisani26a.html
+**Indexed from:** pmlr:v318
+
+---
+
+## Abstract
+
+Machine Reading Comprehension (MRC) systems struggle when user questions are unanswerable given the passage: most simply output “no answer”, leaving users without guidance on how to recover useful information. We introduce a \textit{cause-conditioned multi-task learning (MTL)} framework that turns failure into follow-up by jointly (1) classifying an input as answerable or as one of six fine-grained unanswerability causes (Entity Swap, Number Swap, Antonym, Negation, Mutual Exclusion, No Information), and (2) generating a revised, context-grounded answerable question conditioned on the predicted cause label and an extracted guidance sentence. Using an ensemble of strong readers plus LLMs-as-judges, we apply majority voting to test whether rewrites become answerable. A human study further assesses fluency, relevance, and usefulness. Our cause-conditioning MTL framework yields better recovery from unanswerable inputs and earns strong human ratings, advancing user-supportive, failure-aware MRC.

docs/papers/2026_From_Hints_to_Answers_Uncertainty-Aware_LLM-Guided_Retrieval_for_Multi-Hop_Question_Answering.md

@@ -0,0 +1,13 @@
+# From Hints to Answers: Uncertainty-Aware LLM-Guided Retrieval for Multi-Hop Question Answering
+
+**Authors:** Mahdiyar Ali Akbar Alavi, Bita Azad, Julien Serbanescu, Fattane Zarrinkalam, Faezeh Ensan
+**Venue:** Proceedings of the The 39th Canadian Conference on Artificial Intelligence
+**Year:** 2026
+**URL:** https://proceedings.mlr.press/v318/alavi26a.html
+**Indexed from:** pmlr:v318
+
+---
+
+## Abstract
+
+We propose Generate-Retrieve-Generate (GReG), a training-free pipeline for multi-hop open-domain question answering. GReG uses a strong LLM to generate multiple long-form “hints” that expose implicit intermediate facts in the question, and uses the selected hint as a retrieval query for gathering supporting evidence. To choose among candidate hints, we introduce an uncertainty-aware selection method, which favors lower-entropy generations. By improving retrieval quality, GReG enables a smaller, cost-efficient answer generator to answer complex multi-hop questions more accurately. Experiments on HotpotQA and 2WikiMultihopQA show that GReG achieves state-of-the-art performance under identical retrieval and generation settings.

docs/papers/arxiv_2510.21983v1_Uncovering_the_Persuasive_Fingerprint_of_LLMs_in_Jailbreaking_Attacks.md

@@ -1,15 +0,0 @@
-# Uncovering the Persuasive Fingerprint of LLMs in Jailbreaking Attacks
-
-**Authors:** Havva Alizadeh Noughabi, Julien Serbanescu, Fattane Zarrinkalam, Ali Dehghantanha
-**arXiv ID:** 2510.21983v1
-**Published:** 2025-10-24
-**Updated:** 2025-10-24
-**Categories:** cs.CL, cs.AI
-**PDF:** https://arxiv.org/pdf/2510.21983v1
-**Abstract URL:** https://arxiv.org/abs/2510.21983v1
-
----
-
-## Abstract
-
-Despite recent advances, Large Language Models remain vulnerable to jailbreak attacks that bypass alignment safeguards and elicit harmful outputs. While prior research has proposed various attack strategies differing in human readability and transferability, little attention has been paid to the linguistic and psychological mechanisms that may influence a model's susceptibility to such attacks. In this paper, we examine an interdisciplinary line of research that leverages foundational theories of persuasion from the social sciences to craft adversarial prompts capable of circumventing alignment constraints in LLMs. Drawing on well-established persuasive strategies, we hypothesize that LLMs, having been trained on large-scale human-generated text, may respond more compliantly to prompts with persuasive structures. Furthermore, we investigate whether LLMs themselves exhibit distinct persuasive fingerprints that emerge in their jailbreak responses. Empirical evaluations across multiple aligned LLMs reveal that persuasion-aware prompts significantly bypass safeguards, demonstrating their potential to induce jailbreak behaviors. This work underscores the importance of cross-disciplinary insight in addressing the evolving challenges of LLM safety. The code and data are available.

docs/pdfs/resume_julien_current.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7796f0cfee53567c2e5ea51072794c58cb36a6eec1e2fd82aa84f4aa997e9362
+size 234131

download_publications.py

@@ -0,0 +1,525 @@
+#!/usr/bin/env python3
+"""
+Multi-source publication downloader for the Julien Serbanescu RAG system.
+
+Supersedes download_papers.py, which only knew about arXiv and therefore missed
+every paper published in a venue that does not mirror to arXiv (ACM DL, PMLR).
+
+Sources, in the order they are merged:
+
+    openalex   Broad cross-venue coverage with abstracts. Picks up new papers
+               automatically a few weeks after publication. Primary source.
+    crossref   Authoritative author order and venue for anything with a DOI
+               (ACM, IEEE, Springer). Crossref has no abstracts for most ACM
+               records, so it mainly corrects and enriches.
+    pmlr       Proceedings volumes (Canadian AI, ICML, AISTATS...). PMLR papers
+               are not indexed by OpenAlex for months, so they are scraped from
+               the volume index pages listed in sources.json.
+    arxiv      Preprints.
+    manual     Escape hatch in sources.json for anything all of the above miss.
+
+Records are keyed on a normalised title so the same paper arriving from several
+sources merges into one document instead of duplicating. Later sources fill in
+blank fields but never overwrite a field an earlier source already populated,
+except where a source is explicitly more authoritative (Crossref for authors).
+
+Usage:
+    python download_publications.py
+    python download_publications.py --source openalex --source pmlr
+    python download_publications.py --dry-run
+"""
+
+import argparse
+import html
+import json
+import os
+import re
+import sys
+import time
+import urllib.parse
+import urllib.request
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+DEFAULT_CONFIG = "sources.json"
+DEFAULT_OUTPUT = "docs/papers"
+USER_AGENT = "julienserbanescu-rag/1.0 (publication indexer)"
+ALL_SOURCES = ["openalex", "crossref", "pmlr", "arxiv", "manual"]
+
+
+# ----------------------------------------------------------------------------
+# helpers
+# ----------------------------------------------------------------------------
+
+def load_config(path):
+    with open(path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def http_get(url, timeout=45, retries=3, backoff=2.0):
+    """GET with retries. Returns bytes, or None if every attempt failed."""
+    for attempt in range(1, retries + 1):
+        try:
+            req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
+            with urllib.request.urlopen(req, timeout=timeout) as resp:
+                return resp.read()
+        except Exception as e:
+            if attempt == retries:
+                print(f"    request failed after {retries} attempts: {e}")
+                return None
+            wait = backoff ** attempt
+            print(f"    attempt {attempt} failed ({e}); retrying in {wait:.0f}s")
+            time.sleep(wait)
+    return None
+
+
+def norm_title(title):
+    """Normalise a title into a dedupe key: lowercase alphanumerics only."""
+    return re.sub(r"[^a-z0-9]", "", (title or "").lower())
+
+
+def clean_filename(text):
+    cleaned = re.sub(r"[^\w\-_.]", "_", text)
+    cleaned = re.sub(r"_+", "_", cleaned)
+    return cleaned.strip("_")[:120]
+
+
+def strip_tags(s):
+    return html.unescape(re.sub(r"<[^>]+>", "", s or "")).replace("\xa0", " ").strip()
+
+
+def is_our_author(names, cfg):
+    """True if this author list contains our author.
+
+    Both the family name AND a known given name must appear on the same entry.
+    Serbanescu is a common Romanian surname, and a family-name-only match pulls
+    in unrelated researchers (Crossref returns several).
+    """
+    family = cfg["author"]["family_name"].lower()
+    givens = [g.lower() for g in cfg["author"]["given_names"]]
+    for n in names:
+        low = (n or "").lower()
+        if family in low and any(g in low for g in givens):
+            return True
+    return False
+
+
+def merge_record(store, rec):
+    """Merge a record into the store, keyed on normalised title."""
+    key = norm_title(rec.get("title"))
+    if not key:
+        return
+    if key not in store:
+        store[key] = rec
+        return
+    existing = store[key]
+    for field, value in rec.items():
+        if field == "sources":
+            continue
+        if value and not existing.get(field):
+            existing[field] = value
+    existing.setdefault("sources", [])
+    for s in rec.get("sources", []):
+        if s not in existing["sources"]:
+            existing["sources"].append(s)
+
+
+# ----------------------------------------------------------------------------
+# sources
+# ----------------------------------------------------------------------------
+
+def fetch_openalex(cfg):
+    """OpenAlex: broad coverage, includes abstracts as an inverted index."""
+    print("\n  [openalex] querying...")
+    name = cfg["author"]["full_name"]
+    email = cfg["author"].get("contact_email") or ""
+    params = urllib.parse.urlencode({
+        "filter": f"raw_author_name.search:{name}",
+        "per-page": 100,
+        "mailto": email,
+    })
+    raw = http_get(f"https://api.openalex.org/works?{params}")
+    if not raw:
+        return []
+
+    try:
+        data = json.loads(raw)
+    except Exception as e:
+        print(f"    could not parse OpenAlex response: {e}")
+        return []
+
+    out = []
+    for w in data.get("results", []):
+        authors = [
+            (a.get("author") or {}).get("display_name", "")
+            for a in w.get("authorships", [])
+        ]
+        if not is_our_author(authors, cfg):
+            continue
+
+        # OpenAlex ships abstracts as {word: [positions]}; rebuild the text.
+        abstract = ""
+        inv = w.get("abstract_inverted_index")
+        if inv:
+            positions = {}
+            for word, idxs in inv.items():
+                for i in idxs:
+                    positions[i] = word
+            abstract = " ".join(positions[i] for i in sorted(positions))
+
+        loc = (w.get("primary_location") or {})
+        venue = (loc.get("source") or {}).get("display_name")
+
+        out.append({
+            "title": strip_tags(w.get("title")),
+            "authors": [a for a in authors if a],
+            "abstract": abstract,
+            "venue": venue,
+            "year": w.get("publication_year"),
+            "doi": (w.get("doi") or "").replace("https://doi.org/", "") or None,
+            "url": loc.get("landing_page_url") or w.get("doi"),
+            "sources": ["openalex"],
+        })
+
+    print(f"    matched {len(out)} publication(s)")
+    return out
+
+
+def fetch_crossref(cfg):
+    """Crossref: authoritative author order and venue for DOI-bearing papers."""
+    print("\n  [crossref] querying...")
+    name = cfg["author"]["full_name"]
+    email = cfg["author"].get("contact_email") or ""
+    params = urllib.parse.urlencode({
+        "query.author": name,
+        "rows": 100,
+        "mailto": email,
+    })
+    raw = http_get(f"https://api.crossref.org/works?{params}")
+    if not raw:
+        return []
+
+    try:
+        items = json.loads(raw)["message"]["items"]
+    except Exception as e:
+        print(f"    could not parse Crossref response: {e}")
+        return []
+
+    out = []
+    for w in items:
+        authors = [
+            f"{a.get('given','')} {a.get('family','')}".strip()
+            for a in (w.get("author") or [])
+        ]
+        if not is_our_author(authors, cfg):
+            continue
+
+        container = w.get("container-title") or []
+        issued = (w.get("issued") or {}).get("date-parts") or [[None]]
+
+        out.append({
+            "title": strip_tags((w.get("title") or [""])[0]),
+            # Crossref preserves published author order; treat it as canonical.
+            "authors_authoritative": authors,
+            "authors": authors,
+            "abstract": strip_tags(w.get("abstract")) or "",
+            "venue": container[0] if container else None,
+            "year": issued[0][0],
+            "doi": w.get("DOI"),
+            "url": f"https://doi.org/{w['DOI']}" if w.get("DOI") else None,
+            "sources": ["crossref"],
+        })
+
+    print(f"    matched {len(out)} publication(s)")
+    return out
+
+
+def fetch_pmlr(cfg):
+    """Scrape configured PMLR volume index pages, then each paper's abs page.
+
+    PMLR proceedings take months to reach OpenAlex, so without this the most
+    recent conference papers are invisible to the RAG.
+    """
+    volumes = (cfg.get("pmlr") or {}).get("volumes") or []
+    if not volumes:
+        print("\n  [pmlr] no volumes configured, skipping")
+        return []
+
+    out = []
+    for vol in volumes:
+        print(f"\n  [pmlr] scanning volume {vol}...")
+        raw = http_get(f"https://proceedings.mlr.press/{vol}/")
+        if not raw:
+            continue
+        page = raw.decode("utf-8", errors="replace")
+
+        # Each entry is a <div class="paper"> block; split rather than regex the
+        # whole block, since the markup nests inconsistently.
+        blocks = page.split('<div class="paper">')[1:]
+        matched = 0
+        for block in blocks:
+            authors_m = re.search(r'<span class="authors">(.*?)</span>', block, re.S)
+            if not authors_m:
+                continue
+            authors = [strip_tags(a) for a in authors_m.group(1).split("&nbsp;,")]
+            if len(authors) == 1:
+                authors = [strip_tags(a) for a in re.split(r",\s*", authors_m.group(1))]
+            authors = [a for a in (strip_tags(x) for x in authors) if a]
+            if not is_our_author(authors, cfg):
+                continue
+
+            title_m = re.search(r'<p class="title">(.*?)</p>', block, re.S)
+            info_m = re.search(r'<span class="info">(.*?)</span>', block, re.S)
+            abs_m = re.search(r'href="(https://proceedings\.mlr\.press/[^"]+\.html)"', block)
+
+            title = strip_tags(title_m.group(1)) if title_m else ""
+            info = strip_tags(info_m.group(1)) if info_m else ""
+            abs_url = abs_m.group(1) if abs_m else None
+
+            year = None
+            ym = re.search(r"(20\d{2})", info)
+            if ym:
+                year = int(ym.group(1))
+            if not year:
+                vm = re.search(r"(\d{2})[a-z]$", (abs_url or "").rsplit("/", 1)[-1].split(".")[0])
+                if vm:
+                    year = 2000 + int(vm.group(1))
+
+            abstract = ""
+            if abs_url:
+                time.sleep(1)  # be polite to PMLR
+                abs_raw = http_get(abs_url)
+                if abs_raw:
+                    abs_page = abs_raw.decode("utf-8", errors="replace")
+                    am = re.search(
+                        r'<div[^>]*class="abstract"[^>]*>(.*?)</div>', abs_page, re.S
+                    )
+                    if am:
+                        abstract = strip_tags(am.group(1))
+
+            venue = re.sub(r",\s*PMLR.*$", "", info).strip() or f"PMLR {vol}"
+
+            out.append({
+                "title": title,
+                "authors": authors,
+                "abstract": abstract,
+                "venue": venue,
+                "year": year,
+                "url": abs_url,
+                "sources": [f"pmlr:{vol}"],
+            })
+            matched += 1
+
+        print(f"    matched {matched} publication(s) in {vol}")
+
+    return out
+
+
+def fetch_arxiv(cfg):
+    """arXiv preprints by author."""
+    print("\n  [arxiv] querying...")
+    name = cfg["author"]["full_name"]
+    params = urllib.parse.urlencode({
+        "search_query": f'au:"{name}"',
+        "start": 0,
+        "max_results": 100,
+        "sortBy": "submittedDate",
+        "sortOrder": "descending",
+    })
+    raw = http_get(f"http://export.arxiv.org/api/query?{params}")
+    if not raw:
+        return []
+
+    ns = {"atom": "http://www.w3.org/2005/Atom"}
+    try:
+        root = ET.fromstring(raw)
+    except Exception as e:
+        print(f"    could not parse arXiv response: {e}")
+        return []
+
+    out = []
+    for entry in root.findall("atom:entry", ns):
+        authors = [a.findtext("atom:name", "", ns) for a in entry.findall("atom:author", ns)]
+        if not is_our_author(authors, cfg):
+            continue
+
+        arxiv_id = ""
+        pdf_url = ""
+        for link in entry.findall("atom:link", ns):
+            href = link.get("href", "")
+            if link.get("title") == "pdf":
+                pdf_url = href
+            elif "/abs/" in href:
+                arxiv_id = href.split("/abs/")[-1]
+
+        out.append({
+            "title": " ".join((entry.findtext("atom:title", "", ns) or "").split()),
+            "authors": authors,
+            "abstract": " ".join((entry.findtext("atom:summary", "", ns) or "").split()),
+            "venue": "arXiv preprint",
+            "year": int((entry.findtext("atom:published", "", ns) or "0000")[:4] or 0) or None,
+            "arxiv_id": arxiv_id,
+            "pdf_url": pdf_url,
+            "url": f"https://arxiv.org/abs/{arxiv_id}" if arxiv_id else None,
+            "sources": ["arxiv"],
+        })
+
+    print(f"    matched {len(out)} publication(s)")
+    return out
+
+
+def fetch_manual(cfg):
+    """Entries hand-written into sources.json."""
+    entries = [
+        e for e in (cfg.get("manual_publications") or [])
+        if e.get("title") and not e.get("_example")
+    ]
+    print(f"\n  [manual] {len(entries)} entry(ies) from config")
+    out = []
+    for e in entries:
+        rec = {k: v for k, v in e.items() if not k.startswith("_")}
+        rec["sources"] = ["manual"]
+        out.append(rec)
+    return out
+
+
+FETCHERS = {
+    "openalex": fetch_openalex,
+    "crossref": fetch_crossref,
+    "pmlr": fetch_pmlr,
+    "arxiv": fetch_arxiv,
+    "manual": fetch_manual,
+}
+
+
+# ----------------------------------------------------------------------------
+# output
+# ----------------------------------------------------------------------------
+
+def render_markdown(rec):
+    authors = rec.get("authors_authoritative") or rec.get("authors") or []
+    lines = [f"# {rec.get('title','Untitled')}", ""]
+
+    if authors:
+        lines.append(f"**Authors:** {', '.join(authors)}")
+    if rec.get("venue"):
+        lines.append(f"**Venue:** {rec['venue']}")
+    if rec.get("year"):
+        lines.append(f"**Year:** {rec['year']}")
+    if rec.get("doi"):
+        lines.append(f"**DOI:** {rec['doi']}")
+    if rec.get("arxiv_id"):
+        lines.append(f"**arXiv ID:** {rec['arxiv_id']}")
+    if rec.get("url"):
+        lines.append(f"**URL:** {rec['url']}")
+    if rec.get("pdf_url"):
+        lines.append(f"**PDF:** {rec['pdf_url']}")
+    if rec.get("sources"):
+        lines.append(f"**Indexed from:** {', '.join(rec['sources'])}")
+
+    lines += ["", "---", "", "## Abstract", ""]
+    lines.append(rec.get("abstract") or "(No abstract available from any source.)")
+    lines.append("")
+    return "\n".join(lines)
+
+
+def write_publications(records, output_dir, dry_run=False):
+    out_path = Path(output_dir)
+    if not dry_run:
+        out_path.mkdir(parents=True, exist_ok=True)
+
+    # Rewrite the directory from scratch so renamed or retracted papers do not
+    # linger as stale documents in the index.
+    existing = set()
+    if out_path.exists():
+        existing = {p.name for p in out_path.glob("*.md")}
+
+    written = set()
+    for rec in records:
+        year = rec.get("year") or "n.d."
+        fname = clean_filename(f"{year}_{rec.get('title','untitled')}") + ".md"
+        written.add(fname)
+        if dry_run:
+            continue
+        (out_path / fname).write_text(render_markdown(rec), encoding="utf-8")
+
+    stale = existing - written
+    if not dry_run:
+        for name in stale:
+            (out_path / name).unlink()
+
+    return written, stale
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Download publications from all sources")
+    parser.add_argument("--config", default=DEFAULT_CONFIG)
+    parser.add_argument("--output-dir", default=DEFAULT_OUTPUT)
+    parser.add_argument("--source", action="append", choices=ALL_SOURCES,
+                        help="Limit to specific sources (repeatable). Default: all.")
+    parser.add_argument("--dry-run", action="store_true",
+                        help="Report what would be written without touching disk")
+    parser.add_argument("--min-expected", type=int, default=0,
+                        help="Fail with exit 1 if fewer than N publications are found")
+    args = parser.parse_args()
+
+    if not os.path.exists(args.config):
+        print(f"ERROR: config not found: {args.config}")
+        return 1
+    cfg = load_config(args.config)
+
+    sources = args.source or ALL_SOURCES
+    print("=" * 64)
+    print(f"  Publication download for {cfg['author']['full_name']}")
+    print(f"  Sources: {', '.join(sources)}")
+    print("=" * 64)
+
+    store = {}
+    failed_sources = []
+    for name in sources:
+        try:
+            for rec in FETCHERS[name](cfg):
+                merge_record(store, rec)
+        except Exception as e:
+            print(f"    [{name}] unexpected error: {e}")
+            failed_sources.append(name)
+
+    records = sorted(
+        store.values(),
+        key=lambda r: (r.get("year") or 0, r.get("title") or ""),
+        reverse=True,
+    )
+
+    print("\n" + "=" * 64)
+    print(f"  {len(records)} unique publication(s) after dedupe")
+    print("=" * 64)
+    for r in records:
+        abstract_flag = "abstract" if r.get("abstract") else "NO ABSTRACT"
+        print(f"  {r.get('year') or '????'}  {(r.get('title') or '')[:66]}")
+        print(f"          {r.get('venue') or 'unknown venue'}  [{abstract_flag}]"
+              f"  via {', '.join(r.get('sources', []))}")
+
+    written, stale = write_publications(records, args.output_dir, args.dry_run)
+
+    if args.dry_run:
+        print(f"\n  DRY RUN: would write {len(written)} file(s) to {args.output_dir}/")
+        if stale:
+            print(f"  DRY RUN: would remove {len(stale)} stale file(s): {', '.join(sorted(stale))}")
+    else:
+        print(f"\n  Wrote {len(written)} file(s) to {args.output_dir}/")
+        if stale:
+            print(f"  Removed {len(stale)} stale file(s): {', '.join(sorted(stale))}")
+
+    if failed_sources:
+        print(f"\n  WARNING: these sources errored and contributed nothing: {', '.join(failed_sources)}")
+
+    if args.min_expected and len(records) < args.min_expected:
+        print(f"\n  ERROR: found {len(records)} publication(s), expected at least {args.min_expected}.")
+        print("  Refusing to report success; the index would lose content.")
+        return 1
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

refresh_index.py

@@ -0,0 +1,272 @@
+#!/usr/bin/env python3
+"""
+Knowledge Base Refresh
+
+Re-downloads every source, rebuilds the FAISS index, and optionally deploys to
+the Hugging Face Space.
+
+Safety model
+------------
+The previous version ran every step, ignored failures, and rebuilt the index
+regardless. If GitHub rate-limited a run, the index was silently rebuilt from a
+half-empty docs/ tree and the RAG lost content with no visible error.
+
+This version:
+  * treats each source as independent and records its outcome,
+  * refuses to rebuild if a source failed, unless --allow-partial is given,
+  * refuses to rebuild if the corpus shrank more than --max-shrink percent
+    against the last recorded run,
+  * never pushes a build it would not have rebuilt.
+
+Usage:
+    python refresh_index.py                     # download + rebuild
+    python refresh_index.py --dry-run           # report only, touch nothing
+    python refresh_index.py --skip-download     # rebuild from existing docs
+    python refresh_index.py --push              # rebuild and deploy to HF
+    python refresh_index.py --only publications # run a single source
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+CONFIG_PATH = "sources.json"
+STATE_PATH = ".refresh_state.json"
+
+# Directories whose file count is a proxy for corpus health.
+CORPUS_DIRS = {
+    "publications": ("docs/papers", "*.md"),
+    "readmes": ("docs/readmes", "*.md"),
+    "github_activity": ("docs/github_activity", "*.md"),
+    "pdfs": ("docs/pdfs", "*.pdf"),
+}
+
+
+def load_config():
+    if not os.path.exists(CONFIG_PATH):
+        print(f"ERROR: {CONFIG_PATH} not found.")
+        sys.exit(1)
+    with open(CONFIG_PATH, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def corpus_counts():
+    return {
+        name: len(list(Path(d).glob(pat))) if Path(d).exists() else 0
+        for name, (d, pat) in CORPUS_DIRS.items()
+    }
+
+
+def load_state():
+    if os.path.exists(STATE_PATH):
+        try:
+            with open(STATE_PATH, "r", encoding="utf-8") as f:
+                return json.load(f)
+        except Exception:
+            pass
+    return {}
+
+
+def save_state(state):
+    with open(STATE_PATH, "w", encoding="utf-8") as f:
+        json.dump(state, f, indent=2)
+
+
+def run_step(cmd, label, dry_run=False):
+    """Run one pipeline step. Returns True on success."""
+    print(f"\n{'='*64}\n  {label}\n{'='*64}")
+    if dry_run:
+        print(f"  DRY RUN: would execute {' '.join(str(c) for c in cmd)}")
+        return True
+
+    start = time.time()
+    result = subprocess.run(cmd)
+    elapsed = time.time() - start
+
+    if result.returncode != 0:
+        print(f"\n  FAILED: {label} exited {result.returncode} after {elapsed:.0f}s")
+        return False
+    print(f"\n  OK: {label} ({elapsed:.0f}s)")
+    return True
+
+
+def validate_github_token(token):
+    """Return the token if it authenticates, else empty string."""
+    if not token:
+        return ""
+    try:
+        import requests
+        resp = requests.get(
+            "https://api.github.com/user",
+            headers={"Authorization": f"token {token}"},
+            timeout=10,
+        )
+        if resp.status_code == 401:
+            print("  WARNING: GITHUB_TOKEN is expired or invalid; continuing unauthenticated.")
+            print("  (Unauthenticated GitHub API is rate-limited to 60 requests/hour.)")
+            return ""
+        return token
+    except Exception as e:
+        print(f"  WARNING: could not validate GITHUB_TOKEN ({e}); using it anyway.")
+        return token
+
+
+def push_to_hf(dry_run=False):
+    print(f"\n{'='*64}\n  Deploying to Hugging Face Space\n{'='*64}")
+    if dry_run:
+        print("  DRY RUN: would git add docs/, commit, and push")
+        return True
+
+    subprocess.run(["git", "add", "docs/"])
+    if subprocess.run(["git", "diff", "--staged", "--quiet"]).returncode == 0:
+        print("  No changes to commit; the deployed index is already current.")
+        return True
+
+    stamp = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+    subprocess.run(["git", "commit", "-m", f"chore: refresh knowledge base ({stamp})"])
+    if subprocess.run(["git", "push"]).returncode != 0:
+        print("  ERROR: git push failed. Check your Hugging Face credentials.")
+        return False
+
+    print("  Pushed. The Space will rebuild automatically.")
+    return True
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Refresh the RAG knowledge base")
+    parser.add_argument("--github-token", default=os.getenv("GITHUB_TOKEN", ""))
+    parser.add_argument("--skip-download", action="store_true",
+                        help="Rebuild the index from whatever is already in docs/")
+    parser.add_argument("--only", action="append",
+                        choices=["publications", "readmes", "activity", "resume"],
+                        help="Run only these sources (repeatable)")
+    parser.add_argument("--push", action="store_true",
+                        help="Commit and push to the Hugging Face Space after a clean rebuild")
+    parser.add_argument("--dry-run", action="store_true",
+                        help="Report the plan without executing anything")
+    parser.add_argument("--allow-partial", action="store_true",
+                        help="Rebuild even if a source failed (risks losing content)")
+    parser.add_argument("--max-shrink", type=float, default=25.0,
+                        help="Abort if the corpus shrinks more than this percent (default 25)")
+    args = parser.parse_args()
+
+    cfg = load_config()
+    state = load_state()
+    before = corpus_counts()
+
+    print("=" * 64)
+    print("  RAG Knowledge Base Refresh")
+    print(f"  Started {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
+    print("=" * 64)
+    print("\n  Corpus before:")
+    for k, v in before.items():
+        print(f"    {k:<16} {v}")
+
+    results = {}
+    py = sys.executable
+
+    if not args.skip_download:
+        wanted = args.only or ["publications", "readmes", "activity", "resume"]
+        token = validate_github_token(args.github_token)
+        token_args = ["--token", token] if token else []
+        username = cfg["github"]["username"]
+
+        if "publications" in wanted:
+            results["publications"] = run_step(
+                [py, "download_publications.py", "--min-expected", "1"],
+                "Downloading publications (OpenAlex, Crossref, PMLR, arXiv)",
+                args.dry_run,
+            )
+
+        if "readmes" in wanted:
+            results["readmes"] = run_step(
+                [py, "download_readmes.py", "--username", username,
+                 "--exclude-tpoze"] + token_args,
+                "Downloading GitHub READMEs",
+                args.dry_run,
+            )
+
+        if "activity" in wanted:
+            results["activity"] = run_step(
+                [py, "download_github_activity.py", "--username", username] + token_args,
+                "Downloading GitHub issues and PRs",
+                args.dry_run,
+            )
+
+        if "resume" in wanted:
+            results["resume"] = run_step(
+                [py, "sync_resume.py"],
+                "Syncing current resume",
+                args.dry_run,
+            )
+    else:
+        print("\n  Skipping downloads (--skip-download)")
+
+    failed = [k for k, ok in results.items() if not ok]
+    after = corpus_counts()
+
+    print(f"\n{'='*64}\n  Source results\n{'='*64}")
+    for name in sorted(results):
+        delta = after.get(name, 0) - before.get(name, 0)
+        sign = f"{delta:+d}" if delta else "no change"
+        print(f"    {name:<16} {'OK' if results[name] else 'FAILED':<8} ({sign})")
+
+    # Guard 1: a failed source means the corpus may be incomplete.
+    if failed and not args.allow_partial:
+        print(f"\n  ABORTING: these sources failed: {', '.join(failed)}")
+        print("  Rebuilding now would bake the missing content into the index.")
+        print("  Fix the cause, or re-run with --allow-partial to override.")
+        return 1
+
+    # Guard 2: a large drop usually means an API returned an empty result set.
+    shrink_problems = []
+    for name, prev in (state.get("corpus") or {}).items():
+        now = after.get(name, 0)
+        if prev > 0 and now < prev:
+            pct = (prev - now) / prev * 100
+            if pct > args.max_shrink:
+                shrink_problems.append(f"{name}: {prev} -> {now} ({pct:.0f}% smaller)")
+
+    if shrink_problems and not args.allow_partial:
+        print(f"\n  ABORTING: corpus shrank more than {args.max_shrink:.0f}%:")
+        for p in shrink_problems:
+            print(f"    {p}")
+        print("  This usually means a source returned nothing rather than that content was removed.")
+        print("  Re-run with --allow-partial if the shrinkage is intentional.")
+        return 1
+
+    ok = run_step([py, "build_index.py"], "Rebuilding FAISS index", args.dry_run)
+    if not ok:
+        print("\n  Index rebuild failed. The previously deployed index is untouched.")
+        return 1
+
+    if not args.dry_run:
+        state["corpus"] = after
+        state["last_run"] = datetime.now(timezone.utc).isoformat()
+        save_state(state)
+
+    if args.push:
+        if not push_to_hf(args.dry_run):
+            return 1
+
+    print(f"\n{'='*64}")
+    print("  Refresh complete.")
+    for k, v in after.items():
+        print(f"    {k:<16} {v}")
+    if not args.push:
+        print("\n  Run with --push to deploy to the Hugging Face Space.")
+    print("=" * 64)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scheduled_refresh.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+#
+# Scheduled knowledge-base refresh, intended for cron on an always-on box.
+#
+# Wraps refresh_index.py with the things cron needs and a bare script does not:
+# an absolute working directory, the venv on PATH, a lock so overlapping runs
+# cannot corrupt the index, and a rotating log.
+#
+# Install (on julienlab):
+#     crontab -e
+#     0 3 * * 1  /home/julien/julienserbanescu-rag/scheduled_refresh.sh
+#
+# See AUTOMATION.md for full setup.
+
+set -euo pipefail
+
+REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VENV="${REPO_DIR}/venv"
+LOG_DIR="${REPO_DIR}/logs"
+LOCK="${REPO_DIR}/.refresh.lock"
+LOG="${LOG_DIR}/refresh-$(date -u +%Y%m%d-%H%M%S).log"
+
+mkdir -p "${LOG_DIR}"
+
+# Refuse to start if a previous run is still going. flock releases the lock
+# automatically if the holding process dies, so a crashed run cannot wedge this.
+exec 9>"${LOCK}"
+if ! flock -n 9; then
+    echo "$(date -u +%FT%TZ) another refresh is already running; exiting." >> "${LOG}"
+    exit 0
+fi
+
+cd "${REPO_DIR}"
+
+if [ -x "${VENV}/bin/python" ]; then
+    PYTHON="${VENV}/bin/python"
+else
+    PYTHON="$(command -v python3)"
+    echo "WARNING: no venv at ${VENV}; falling back to ${PYTHON}" | tee -a "${LOG}"
+fi
+
+{
+    echo "=============================================================="
+    echo " Scheduled refresh started $(date -u +%FT%TZ)"
+    echo " Repo:   ${REPO_DIR}"
+    echo " Python: ${PYTHON}"
+    echo "=============================================================="
+} >> "${LOG}"
+
+status=0
+"${PYTHON}" refresh_index.py --push >> "${LOG}" 2>&1 || status=$?
+
+{
+    echo "--------------------------------------------------------------"
+    if [ "${status}" -eq 0 ]; then
+        echo " Finished OK $(date -u +%FT%TZ)"
+    else
+        echo " FAILED (exit ${status}) $(date -u +%FT%TZ)"
+        echo " The previously deployed index was left untouched."
+    fi
+} >> "${LOG}"
+
+# Keep the last 12 runs.
+ls -1t "${LOG_DIR}"/refresh-*.log 2>/dev/null | tail -n +13 | xargs -r rm -f
+
+exit "${status}"

sources.json

@@ -0,0 +1,43 @@
+{
+  "_comment": "Central config for the knowledge-base refresh pipeline. Edit this rather than the scripts.",
+
+  "author": {
+    "full_name": "Julien Serbanescu",
+    "given_names": ["Julien"],
+    "family_name": "Serbanescu",
+    "openalex_author_id": null,
+    "contact_email": "julien.serbanescu@gmail.com"
+  },
+
+  "github": {
+    "username": "Julien-ser",
+    "exclude_forks": true
+  },
+
+  "pmlr": {
+    "_comment": "PMLR volumes to scan. New proceedings appear here before OpenAlex indexes them. Canadian AI 2026 = v318.",
+    "volumes": ["v318"]
+  },
+
+  "resume": {
+    "_comment": "Copied into docs/pdfs on each refresh so the RAG always reflects the current resume. First path that exists wins.",
+    "source_paths": [
+      "C:/Users/a1/Documents/Obsidian Vault/MyExp/resume.pdf",
+      "~/Documents/Obsidian Vault/MyExp/resume.pdf"
+    ],
+    "dest_name": "resume_julien_current.pdf"
+  },
+
+  "manual_publications": [
+    {
+      "_comment": "Escape hatch for anything the APIs miss. Entries here are always included.",
+      "_example": true,
+      "title": null,
+      "authors": [],
+      "venue": null,
+      "year": null,
+      "abstract": null,
+      "url": null
+    }
+  ]
+}

sync_resume.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""
+Copy the current resume PDF into docs/pdfs so the RAG always answers from the
+latest version rather than whichever copy happened to be committed.
+
+Paths are configured under "resume" in sources.json; the first path that exists
+wins, so the same config works on the Windows workstation and on a Linux box.
+
+Usage:
+    python sync_resume.py
+    python sync_resume.py --config sources.json
+"""
+
+import argparse
+import json
+import os
+import shutil
+import sys
+from pathlib import Path
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Sync the current resume into docs/pdfs")
+    parser.add_argument("--config", default="sources.json")
+    parser.add_argument("--dest-dir", default="docs/pdfs")
+    args = parser.parse_args()
+
+    if not os.path.exists(args.config):
+        print(f"  ERROR: config not found: {args.config}")
+        return 1
+
+    with open(args.config, "r", encoding="utf-8") as f:
+        cfg = json.load(f)
+
+    resume_cfg = cfg.get("resume") or {}
+    candidates = resume_cfg.get("source_paths") or []
+    dest_name = resume_cfg.get("dest_name") or "resume_current.pdf"
+
+    src = None
+    for raw in candidates:
+        p = Path(os.path.expanduser(raw))
+        if p.is_file():
+            src = p
+            break
+
+    if src is None:
+        # Not fatal: the pipeline should still rebuild from everything else.
+        print("  No resume found at any configured path; skipping.")
+        for raw in candidates:
+            print(f"    tried: {raw}")
+        return 0
+
+    dest_dir = Path(args.dest_dir)
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    dest = dest_dir / dest_name
+
+    if dest.exists() and dest.stat().st_size == src.stat().st_size:
+        print(f"  Resume already current ({dest_name}, {src.stat().st_size:,} bytes)")
+        return 0
+
+    shutil.copy2(src, dest)
+    print(f"  Copied resume: {src}")
+    print(f"    -> {dest} ({dest.stat().st_size:,} bytes)")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())