graphrag: route Gemini 2.5 via OpenAI-compat endpoint; add index retry wrapper

Native litellm gemini transport dropped connections (Server disconnected / 503) and aborted extract_graph even sequentially; the OpenAI-compatible endpoint (still gemini-2.5-flash) is stable and completed a full index first try at concurrency 6. Add run_graphrag_index.sh: loops graphrag index until entities.parquet exists (cache resumes), since one exhausted-retry call otherwise aborts the whole stage. Calibrated course-content cost ~= $88/1M corpus tokens.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

data/graphrag/settings.yaml

@@ -7,21 +7,31 @@
 
 completion_models:
   default_completion_model:
-    model_provider: gemini # -> litellm "gemini/gemini-2.5-flash" (reads GEMINI_API_KEY)
+    # Still Gemini 2.5 Flash, but routed through Gemini's OpenAI-compatible
+    # endpoint instead of litellm's native gemini transport: the native path
+    # drops connections ("Server disconnected" / 503) under sustained
+    # extraction and aborts the stage; the OpenAI client path is far steadier.
+    model_provider: openai
     model: gemini-2.5-flash
+    api_base: https://generativelanguage.googleapis.com/v1beta/openai
     auth_method: api_key
     api_key: ${GEMINI_API_KEY}
     # graphrag defaults to 25 concurrent requests; that burst triggers Gemini
     # connection resets ([Errno 32] Broken pipe). Keep it low + let rate_limit
     # pace throughput. graphrag caches each successful call, so a re-run resumes.
-    concurrent_requests: 4
+    # Moderate parallelism over the OpenAI-compatible endpoint. Caching + the
+    # index retry-loop absorb any transient drops.
+    concurrent_requests: 6
     async_mode: threaded
     call_args:
       temperature: 0 # deterministic extraction
+    # Gemini throws transient 503 / broken-pipe under load; retry hard. graphrag
+    # caches each success, so a failed run is re-run to resume (see index loop).
     retry:
       type: exponential_backoff
-      max_retries: 10
-      max_delay: 60
+      max_retries: 20
+      base_delay: 2
+      max_delay: 120
     # Throttle to stay friendly to the Gemini free/low tier and avoid the
     # TPM blowups noted in evals.md. On a paid tier-1 key these can be raised
     # (e.g. requests_per_period: 200) to finish the full index faster.

data/scraping_scripts/run_graphrag_index.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Build the GraphRAG index (experiment: GraphRAG vs classical RAG).
+#
+# graphrag aborts the whole extract_graph stage if a single Gemini call fails
+# after its retries (transient 503 / broken pipe happen under load), but it
+# CACHES every successful call -- so re-running resumes and only retries what's
+# missing, at no extra cost. This wrapper loops the index until entities.parquet
+# exists (or MAX_ATTEMPTS is hit).
+#
+# Prereqs: a valid GEMINI_API_KEY (indexing LLM, 2.5 Flash) and OPENAI_API_KEY
+# (embeddings) in .env. Cohere is NOT needed to index. The graph DB is
+# local-only (data/graphrag/output) and never uploaded to the HF bundle.
+#
+# Usage:
+#   data/scraping_scripts/graphrag_prep_input.py --sources full_stack_ai_engineering
+#   bash data/scraping_scripts/run_graphrag_index.sh
+set -u
+
+ROOT="${GRAPHRAG_ROOT:-data/graphrag}"
+MAX_ATTEMPTS="${MAX_ATTEMPTS:-8}"
+OUT="$ROOT/output/entities.parquet"
+
+for i in $(seq 1 "$MAX_ATTEMPTS"); do
+  echo "=== graphrag index attempt $i/$MAX_ATTEMPTS ==="
+  uv run --env-file .env graphrag index --root "$ROOT"
+  if [ -f "$OUT" ]; then
+    echo "SUCCESS: $OUT present after attempt $i"
+    exit 0
+  fi
+  echo "attempt $i did not produce entities.parquet; retrying (cache resumes)"
+  sleep 5
+done
+
+echo "FAILED: no entities.parquet after $MAX_ATTEMPTS attempts" >&2
+exit 1