docs: update README and main.py to reflect two-tier architecture (v2.0.0)

README.md

@@ -10,9 +10,181 @@ pinned: false
 
 # GCAS Excel Search Engine
 
-Natural-language search API over Gujarat College Admissions System (GCAS) data.
+Natural-language search API over Gujarat College Admissions System (GCAS) data. Supports English, Hindi (Hinglish), and Gujarati queries.
 
-**POST** `/search` — query in English, Hindi, or Gujarati
-**GET** `/health` — index status
-**GET** `/schema` — table schemas
-**GET** `/docs` — Swagger UI
+---
+
+## Architecture
+
+The engine runs a **two-tier pipeline** — structured lookup first, semantic fallback second.
+
+**Tier 1 — Structured in-memory lookup (~50–200 ms)**
+Every query is parsed into a `QueryPlan` (college name, district, university, program, gender, category, medium, type, intent). Resolved entities are matched directly against the in-memory data store with O(n) row scanning. This handles ~95% of real queries with `confidence: high`.
+
+**Tier 2 — FAISS semantic fallback**
+Only triggered when Tier 1 produces zero candidates (vague / open-ended queries with no resolved entities). The query is embedded and compared against dense FAISS vectors built over all ~50k rows.
+
+**LLM reranking** is an explicit opt-in (`use_llm_rerank: true`) — it is NOT part of the default path. Default is `false`.
+
+---
+
+## Endpoints
+
+| Method | Path | Auth | Description |
+|--------|------|------|-------------|
+| `GET` | `/health` | No | Liveness check + index status |
+| `GET` | `/schema` | No | Table names, columns, row counts |
+| `POST` | `/search` | Bearer token | Natural-language search |
+| `POST` | `/reindex` | Bearer token | Rebuild FAISS index from Excel folder |
+| `GET` | `/docs` | No | Swagger UI (auto-generated) |
+| `GET` | `/redoc` | No | ReDoc UI |
+
+---
+
+## POST /search
+
+### Request body
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `query` | string | *required* | Query in English, Hindi (Hinglish), or Gujarati |
+| `top_k` | int | 10 | Max results (1–100). Actual count may be lower — the engine auto-caps based on intent (e.g. hostel → 1, fees → 8) |
+| `tables` | list[str] | null | Restrict to specific table name(s). null = all tables |
+| `use_llm_rerank` | bool | **false** | Opt-in LLM reranking pass (adds ~3–10s latency) |
+| `llm_provider` | string | server default | `"openai"` or `"anthropic"` |
+| `llm_model` | string | server default | Model name override |
+| `api_key` | string | server default | API key override for chosen provider |
+
+### Response body
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `query` | string | Original query as submitted |
+| `total_results` | int | Number of results returned |
+| `results` | list[SearchResult] | Ranked result rows |
+| `search_time_ms` | float | End-to-end latency |
+| `reranked` | bool | Whether LLM reranking was applied |
+| `detected_language` | string | `en` / `hi` / `gu` / `unknown` |
+| `corrected_query` | string\|null | Query after alias resolution and entity correction |
+| `entity_corrections` | list[EntityCorrection] | Spelling / ASR corrections applied |
+| `confidence_level` | string | `high` (structured hit) / `medium` / `low` (FAISS fallback) |
+| `detected_intent` | string | `fees` / `hostel` / `cutoff` / `facilities` / `courses` / `naac` / `contact` / `general` |
+| `did_you_mean` | list[string] | Suggestions when confidence is low |
+
+### SearchResult fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `table` | string | Source table (Excel filename stem) |
+| `row_index` | int | Original row index in the Excel file |
+| `score` | float | Relevance score (higher = more relevant) |
+| `llm_reason` | string\|null | LLM explanation (only when `reranked: true`) |
+| `data` | object | Row data filtered to fields relevant to the detected intent |
+
+`data` is **intent-filtered**: only the columns relevant to your query intent are returned, keeping responses compact. For example, a `fees` query returns fee columns only; a `hostel` query returns hostel-specific fields.
+
+---
+
+## Supported Query Types
+
+The engine is designed around the full GCAS query taxonomy:
+
+**College-level queries**
+- "M N College ke baare mein sab batao"
+- "GLS College contact number"
+- "NAAC A grade colleges in Ahmedabad"
+- "Government colleges in Rajkot district"
+- "Grant-in-aid Gujarati medium arts colleges"
+
+**Fees queries**
+- "Gujarat Commerce College ki fees kitni hai?"
+- "B.Com fees in Surat government colleges"
+- "SC category ke liye engineering fees"
+- "Girls ke liye medical college fees in Vadodara"
+
+**Hostel queries**
+- "M N College mein girls hostel hai kya?"
+- "Rajkot mein boys hostel wale colleges"
+- "Engineering colleges with hostel facility in Ahmedabad"
+
+**Cutoff / admission queries**
+- "IMN Law College ka cutoff kya hai?"
+- "GTU ke under OBC category computer science cutoff"
+- "Mehsana district mein arts college cutoff"
+
+**Program / courses queries**
+- "M N College mein evening batch hai kya?"
+- "BCA kahan kahan milta hai Gujarat mein?"
+- "Rajkot mein law college hai kya?"
+- "LLB kitne saal ka course hai?"
+
+**Infrastructure / facilities**
+- "Anand district mein library wale colleges"
+- "NAAC accredited self-financed colleges under GTU"
+
+---
+
+## Authentication
+
+All `POST` endpoints require a Bearer token:
+
+```
+Authorization: Bearer <API_SECRET_TOKEN>
+```
+
+The token is configured via the `API_SECRET_TOKEN` environment variable on the server. If not set, the API runs open (dev mode).
+
+---
+
+## Example (curl)
+
+```bash
+curl -X POST https://tanmay-bm-gsearch-api.hf.space/search \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "query": "GLS College ki fees kitni hai",
+    "top_k": 5
+  }'
+```
+
+```bash
+# With LLM reranking (slower, ~3-10s extra)
+curl -X POST https://tanmay-bm-gsearch-api.hf.space/search \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "query": "engineering colleges in Ahmedabad with hostel and NAAC A grade",
+    "top_k": 5,
+    "use_llm_rerank": true,
+    "llm_provider": "openai",
+    "api_key": "sk-..."
+  }'
+```
+
+---
+
+## Data Tables
+
+| Table | Description |
+|-------|-------------|
+| `CollegeMaster` | College identity, type, district, university, NAAC grade, medium, gender |
+| `CollegeIntakeMaster` | Programs offered, intake seats, fees by category |
+| `CollegeInfrastructure` | Hostel, library, labs, sports, and other facilities |
+| `CutOff_2025-26` | Admission cutoff ranks by program, round, and category |
+
+---
+
+## Multilingual Support
+
+Queries can mix English, Hindi (Hinglish romanised), and Gujarati. The normaliser handles common transliterations:
+
+| Input | Resolved |
+|-------|----------|
+| `ki fees kitni hai` | fees |
+| `ke baare mein sab batao` | (removed) |
+| `hostel milta hai kya` | hostel available |
+| `kitne saal ka course` | how many years course |
+| `kahan milta hai` | where available |
+
+College names, district names, and university names are fuzzy-matched against the database vocabulary (edit distance + phonetic matching), so typos and ASR errors are automatically corrected.

main.py

@@ -10,23 +10,28 @@ Endpoints
   POST /reindex       – rebuild FAISS index from the Excel folder
   GET  /docs          – Swagger UI (auto-generated)
 
+Pipeline
+--------
+  Tier 1: Structured in-memory lookup (~50–200 ms, confidence=high)
+    QueryPlan is built from the query (college, district, university,
+    program, gender, category, intent, etc.) and matched directly against
+    the data store via O(n) row scan.
+  Tier 2: FAISS semantic fallback (triggered only when Tier 1 returns [])
+    Query is embedded and compared against dense FAISS vectors (~50k rows).
+  LLM reranking: opt-in only (use_llm_rerank=true), not on by default.
+
 Startup behaviour
 -----------------
-  On first boot the server tries to load a persisted FAISS cache from
+  On first boot the server tries to load a persisted cache from
   ./index_cache/.  If none exists it indexes the Excel files immediately
   (this takes ~1-3 min for ~50 k rows with the local embedding model).
 
 Usage example (curl)
 --------------------
   curl -X POST http://localhost:8000/search \
+       -H 'Authorization: Bearer <token>' \
        -H 'Content-Type: application/json' \
-       -d '{
-             "query": "engineering colleges in Ahmedabad with hostel and NAAC A grade",
-             "top_k": 5,
-             "use_llm_rerank": true,
-             "llm_provider": "openai",
-             "api_key": "sk-..."
-           }'
+       -d '{"query": "GLS College ki fees kitni hai", "top_k": 5}'
 """
 from __future__ import annotations
 
@@ -117,10 +122,12 @@ async def lifespan(app: FastAPI):
 app = FastAPI(
     title="GCAS Excel Search Engine",
     description=(
-        "Natural-language search API over Gujarat College Admissions System (GCAS) "
-        "Excel data.  Hybrid pipeline: dense retrieval (FAISS) + LLM reranking."
+        "Natural-language search API over Gujarat College Admissions System (GCAS) Excel data. "
+        "Two-tier pipeline: structured in-memory lookup (primary, ~50–200ms) + "
+        "FAISS semantic fallback. Supports English, Hindi (Hinglish), and Gujarati queries. "
+        "LLM reranking is opt-in only."
     ),
-    version="1.0.0",
+    version="2.0.0",
     lifespan=lifespan,
     docs_url="/docs",
     redoc_url="/redoc",
@@ -198,7 +205,7 @@ def search(request: SearchRequest) -> SearchResponse:
     | `query` | string | *required* | Natural language query |
     | `top_k` | int | 10 | Max results to return (1–100) |
     | `tables` | list[str] | null | Restrict to specific table names |
-    | `use_llm_rerank` | bool | true | Enable LLM reranking pass |
+    | `use_llm_rerank` | bool | false | Opt-in LLM reranking (adds 3–10s latency) |
     | `llm_provider` | string | *(server default)* | `"openai"` or `"anthropic"` |
     | `llm_model` | string | *(server default)* | Model name |
     | `api_key` | string | *(server default)* | API key override |