Deploy from sec-dense-fts

README.md

@@ -1,13 +1,170 @@
 ---
 title: SEC FTS Hybrid Comparison
-emoji: 🌍
-colorFrom: purple
-colorTo: purple
+emoji: 🔎
+colorFrom: blue
+colorTo: indigo
 sdk: gradio
 sdk_version: 6.29.0
 python_version: '3.12'
 app_file: app.py
 pinned: false
+short_description: Pinecone full-text search vs dense vectors on 10-Ks
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# SEC Document Search — Pinecone FTS + Semantic
+
+A demo application showing how to build a hybrid search system over SEC 10-K filings using [Pinecone's full-text search](https://docs.pinecone.io/guides/search/full-text-search) combined with dense vector embeddings.
+
+The app supports three search modes:
+
+| Mode | How it works |
+|---|---|
+| **Full-text** | BM25 keyword search over the document text |
+| **Query builder (full-text)** | Construct complex full-text queries — Lucene boolean / phrase / proximity / boost / prefix scoring plus text-match and metadata filters — starting from 10 worked examples |
+| **Semantic** | Dense vector similarity via OpenAI embeddings |
+| **Compare: dense vs full-text** | Runs a dense search and a full-text search side by side and contrasts them — overlap, rank shifts, keyword coverage, latency, and an RRF-fused list |
+| **Hybrid** | Dense vector ranking with a must-contain keyword filter — results are semantically ranked and guaranteed to contain the specified terms |
+
+## Data
+
+`example_data/` contains chunked 10-K filings for six companies across six years:
+
+| Ticker | Company | Years |
+|---|---|---|
+| AAPL | Apple | 2019–2024 |
+| AMZN | Amazon | 2019–2024 |
+| F | Ford | 2019–2024 |
+| GM | General Motors | 2019–2024 |
+| MSFT | Microsoft | 2019–2024 |
+| ORCL | Oracle | 2019–2024 |
+
+~24,000 document chunks total. Each chunk has an `_id`, `text`, `ticker`, `filing_type`, `year`, and `chunk_index`.
+
+## Prerequisites
+
+- Python 3.12+
+- [`uv`](https://docs.astral.sh/uv/)
+- A [Pinecone API key](https://app.pinecone.io/) (free tier works)
+- An [OpenAI API key](https://platform.openai.com/api-keys)
+
+## Setup
+
+```bash
+git clone https://github.com/tim-pinecone/sec-dense-fts
+cd sec-dense-fts
+
+cp .env.example .env
+# Add your PINECONE_API_KEY and OPENAI_API_KEY to .env
+
+uv sync
+```
+
+## Ingest
+
+Creates the Pinecone index, embeds all chunks with `text-embedding-3-small`, and upserts them in batches. Safe to re-run — skips index creation if it already exists.
+
+```bash
+uv run main.py
+```
+
+Ingestion takes a few minutes (OpenAI embedding calls are the bottleneck). The script polls until all documents are searchable before exiting.
+
+## Run the app
+
+Two UIs share the same search logic (`search_core.py`, `fts_queries.py`):
+
+| UI | Command | URL | Notes |
+|---|---|---|---|
+| Gradio | `uv run python app.py` | [http://localhost:7860](http://localhost:7860) | What the Hugging Face Space runs |
+| Streamlit | `uv run streamlit run app_streamlit.py` | [http://localhost:8501](http://localhost:8501) | Local development UI |
+
+Both have the same five tabs/modes. Use the sidebar to filter by ticker and year, pick a search mode, and enter your query — or pick one of the prepared **Example queries** at the top of each mode (three per mode; ten in the query builder). Examples live in `fts_queries.py` (`MODE_EXAMPLES`, `EXAMPLES`).
+
+## Full-text query builder
+
+The **Query builder** mode exposes the full Pinecone FTS query surface. Queries are built in three parts, and the exact `documents.search(...)` request is shown as JSON and Python before running:
+
+1. **Scoring** — either BM25 keywords (`text`) or Lucene (`query_string`). Lucene can be written raw or assembled row-by-row with the clause builder (MUST `+` / MUST NOT `-` / SHOULD, term / phrase / phrase prefix, slop `~N`, boost `^N`).
+2. **Text-match filters** — `$match_phrase`, `$match_all`, `$match_any`, each optionally negated with `$not`, combined with `$and` or `$or`.
+3. **Metadata filters** — ticker `$in` / `$nin`, year range, chunk-index range.
+
+Pick an example from the dropdown to load it into the builder, then tweak it:
+
+| Example | Demonstrates |
+|---|---|
+| Supply-chain shortages, excluding COVID | `text:(("supply chain" OR semiconductor) AND shortage) NOT text:(covid)` |
+| Cyber incidents | `text:(+cybersecurity ransomware^3 breach -insurance)` |
+| Rising interest rates | Proximity: `text:("interest rates increase"~5)` |
+| China trade & tariffs | Term boost: `text:(tariffs^3 trade china)` |
+| AI mentions | Phrase prefix `"artificial intel"*` + required term + year range |
+| EV batteries at Ford & GM | BM25 + `$match_phrase` filter + ticker `$in` + year range |
+| Cloud growth, no pandemic talk | `$not` + `$match_any` exclusion, ticker `$nin` |
+| Regulators: EC or DOJ | `$or` across two `$match_phrase` filters |
+| Inflation & input costs | `$match_all` + year range |
+| Buybacks vs. dividends | Required OR-groups `+(a OR b) +(c OR d)` |
+
+Things the server enforces (surfaced in the UI):
+
+- One scoring type per request — `text` **or** `query_string`, never mixed.
+- `query_string` clauses may not set `fields`; qualify terms inline (`text:(...)`).
+- A Lucene query of only exclusions (`text:(-covid)`) is rejected.
+- Proximity, boost and phrase prefix are scoring-only — they can't be used in `filter`.
+- Phrase-prefix matches all receive the same constant score.
+
+The query compilation logic and examples live in `fts_queries.py`.
+
+## Comparing dense vs full-text
+
+**Compare** mode runs both searches in parallel for the same question:
+
+- **Dense side** — the query is embedded with `text-embedding-3-small` and ranked by cosine similarity.
+- **Full-text side** — either the same query as BM25 keywords, or whatever complex query is currently set up in the **Query builder** (Lucene, text-match filters, metadata filters). When using the builder, you can optionally apply its filters to the dense side too, so only the ranking signal differs.
+
+Sidebar ticker/year filters apply to both sides. The results show:
+
+| View | What it tells you |
+|---|---|
+| Overlap / Jaccard, only-dense, only-full-text | How much the two retrieval methods agree in the top-k |
+| Latency | Dense (embedding + search) vs full-text search time |
+| Keyword coverage | Share of the query's keywords present in each result — dense results with low coverage are paraphrase / concept matches that BM25 can't find |
+| Side by side | Both ranked lists, with badges showing each result's rank in the other list, and keyword highlighting on both |
+| Rank comparison | One table of every retrieved chunk with its dense rank, full-text rank and Δ |
+| Fused (RRF) | Client-side reciprocal rank fusion (k = 60) of the two lists — a preview of a two-query hybrid |
+
+## Deploying to Hugging Face Spaces
+
+The Space runs the **Gradio** app (`app.py`) on free ZeroGPU hardware — no Docker needed. The YAML block at the top of this README is the Space config, and `requirements.txt` holds the Space's Python dependencies (Gradio itself comes from `sdk_version`). ZeroGPU requires at least one `@spaces.GPU` function; `app.py` defines a no-op one since all compute happens in Pinecone and OpenAI.
+
+1. Add a Hugging Face write token to `.env` as `HF_TOKEN`.
+2. Preview what will be uploaded:
+   ```bash
+   uv run python deploy_space.py <owner>/<space-name> --dry-run
+   ```
+3. Deploy. The first run creates the Space; `--set-secrets` copies `PINECONE_API_KEY` and `OPENAI_API_KEY` from `.env` into the Space's secrets (only needed once, or when keys change):
+   ```bash
+   uv run python deploy_space.py <owner>/<space-name> --set-secrets [--private]
+   ```
+
+`deploy_space.py` uploads an explicit allowlist (`README.md`, `requirements.txt`, `app.py`, `search_core.py`, `fts_queries.py`), so `.env`, the example data, and the Streamlit app never leave your machine. Re-run it without `--set-secrets` to push code changes.
+
+## Index schema
+
+```
+text          — full-text search field (BM25, English, stemming enabled)
+embedding     — dense vector, 1536 dims, cosine similarity
+ticker        — filterable metadata (string)
+filing_type   — filterable metadata (string)
+year          — filterable metadata (integer)
+chunk_index   — filterable metadata (integer)
+```
+
+The FTS and vector fields are declared in the schema at index creation. Metadata fields (`ticker`, `filing_type`, `year`, `chunk_index`) are automatically indexed — they do not need to be declared.
+
+## How hybrid search works
+
+The hybrid mode uses a single Pinecone query:
+
+- `score_by` — dense vector cosine similarity (semantic ranking)
+- `filter` — `$match_all` on the text field (hard lexical requirement)
+
+This means results are ordered by semantic relevance, but only chunks that contain all the specified keywords are returned. It's useful for queries like "what does MSFT say about Azure capital expenditure" — the semantic query captures the intent, and the text filter ensures the specific terms are present.

app.py

@@ -0,0 +1,516 @@
+import html
+import json
+import os
+
+import gradio as gr
+import pandas as pd
+
+import fts_queries as fq
+from search_core import (
+    INCLUDE_FIELDS,
+    RRF_K,
+    TICKERS,
+    YEARS,
+    compare_stats,
+    highlight,
+    keyword_coverage,
+    match_record,
+    metadata_filter,
+    missing_env,
+    rank_badge,
+    rank_rows,
+    rrf_fuse,
+    run_compare,
+    run_search_hybrid,
+    run_search_semantic,
+    run_search_text,
+    search,
+)
+
+os.environ["GRADIO_SSR_MODE"] = "False"
+
+try:
+    import spaces
+
+    @spaces.GPU
+    def zero_gpu_noop():
+        return True
+except Exception:
+    def zero_gpu_noop():
+        return False
+
+
+BLANK = "— Start from scratch —"
+N_CLAUSES = 6
+N_TEXT_FILTERS = 4
+ANY_YEAR = "Any"
+BUILDER_EXAMPLES = {ex["name"]: ex for ex in fq.EXAMPLES}
+
+CSS = """
+.card {border: 1px solid var(--border-color-primary); border-radius: 8px; padding: 10px 12px;
+       margin-bottom: 8px; background: var(--block-background-fill);}
+.card .hdr {display: flex; justify-content: space-between; gap: 8px; font-size: 14px;}
+.card .score {font-variant-numeric: tabular-nums; color: var(--body-text-color-subdued);}
+.card .badge {font-size: 12px; color: var(--body-text-color-subdued); margin-top: 2px;}
+.card .snip {font-size: 13px; line-height: 1.45; margin-top: 6px;}
+.card details {font-size: 13px; margin-top: 4px;}
+.card mark, .snip mark {background: rgba(250, 204, 21, .45); color: inherit; padding: 0 1px; border-radius: 2px;}
+.count {font-size: 13px; color: var(--body-text-color-subdued); margin: 4px 0 8px;}
+.err {border: 1px solid #dc2626; border-radius: 8px; padding: 10px 12px; color: #dc2626;}
+.metrics {display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 8px; margin: 4px 0 8px;}
+.metric {border: 1px solid var(--border-color-primary); border-radius: 8px; padding: 8px 10px;}
+.metric .label {font-size: 12px; color: var(--body-text-color-subdued);}
+.metric .value {font-size: 22px; font-weight: 600; font-variant-numeric: tabular-nums;}
+.metric .sub {font-size: 12px; color: var(--body-text-color-subdued);}
+.cols {display: grid; grid-template-columns: 1fr 1fr; gap: 12px;}
+@media (max-width: 800px) {.cols {grid-template-columns: 1fr;}}
+"""
+
+CHEAT_SHEET = """
+| Goal | Lucene (`query_string` scoring) | Filter operator (hard constraint, no scoring) |
+|---|---|---|
+| Any of these terms | `text:(cloud revenue)` | `{"text": {"$match_any": "cloud revenue"}}` |
+| All terms, any order | `text:(+cloud +revenue)` | `{"text": {"$match_all": "cloud revenue"}}` |
+| Exact phrase | `text:("supply chain")` | `{"text": {"$match_phrase": "supply chain"}}` |
+| Exclude | `text:(cloud -covid)` / `NOT text:(covid)` | `{"$not": {"text": {"$match_any": "covid"}}}` |
+| Boolean nesting | `text:((a OR b) AND c)` | `{"$or": [...]}`, `{"$and": [...]}` |
+| Proximity (within N words) | `text:("interest rates increase"~5)` | — scoring only |
+| Boost a term | `text:(tariffs^3 trade)` | — scoring only |
+| Phrase prefix | `text:("artificial intel"*)` (≥ 2 words) | — scoring only |
+
+- One scoring type per request: BM25 `text` **or** `query_string`, never both.
+- Filters run first and shrink the candidate set; scoring ranks what's left.
+- The `text` field uses English stemming, so `increase` also matches `increases` / `increased`.
+- A Lucene query made only of exclusions (`-covid`) is rejected — include at least one positive clause.
+- Phrase-prefix matches all get the same constant score.
+"""
+
+
+def card_html(r: dict, terms: list[str], rank: int | None = None, badge: str | None = None) -> str:
+    snippet, full = highlight(r["text"], terms)
+    prefix = f"<b>#{rank}</b> · " if rank else ""
+    meta = f"{prefix}<b>{html.escape(r['ticker'])}</b> · {r['year']} · {html.escape(r['filing'])} · chunk {r['chunk']}"
+    badge_html = f"<div class='badge'>{html.escape(badge)}</div>" if badge else ""
+    more = f"<details><summary>Full text</summary>{full}</details>" if len(r["text"]) > 300 else ""
+    return (
+        f"<div class='card'><div class='hdr'><span>{meta}</span><span class='score'>{r['score']:.4f}</span></div>"
+        f"{badge_html}<div class='snip'>{snippet}</div>{more}</div>"
+    )
+
+
+def results_html(records: list[dict], terms: list[str]) -> str:
+    if not records:
+        return "<div class='count'>No results found.</div>"
+    cards = "".join(card_html(r, terms) for r in records)
+    return f"<div class='count'>{len(records)} result(s)</div>{cards}"
+
+
+def error_html(message: str) -> str:
+    return f"<div class='err'>{html.escape(message)}</div>"
+
+
+def metric_html(label: str, value: str, sub: str = "") -> str:
+    sub_html = f"<div class='sub'>{html.escape(sub)}</div>" if sub else ""
+    return f"<div class='metric'><div class='label'>{label}</div><div class='value'>{value}</div>{sub_html}</div>"
+
+
+def mode_example(mode: str, name: str, fields: list[str]):
+    ex = next((e for e in fq.MODE_EXAMPLES[mode] if e["name"] == name), None)
+    if not ex:
+        return [gr.update() for _ in fields] + [""]
+    return [ex["values"].get(f, "") for f in fields] + [f"ℹ️ {ex['description']}"]
+
+
+def do_fulltext(query, tickers, years, top_k):
+    if not query.strip():
+        return error_html("Enter a keyword query.")
+    try:
+        resp = run_search_text(query, tickers, years, int(top_k))
+    except Exception as e:
+        return error_html(f"Search failed: {e}")
+    terms = fq.highlight_terms({"scoring": "text", "bm25": query})
+    return results_html([match_record(m) for m in resp.matches], terms)
+
+
+def do_semantic(query, tickers, years, top_k):
+    if not query.strip():
+        return error_html("Enter a semantic query.")
+    try:
+        resp = run_search_semantic(query, tickers, years, int(top_k))
+    except Exception as e:
+        return error_html(f"Search failed: {e}")
+    return results_html([match_record(m) for m in resp.matches], [])
+
+
+def do_hybrid(sem_query, txt_filter, tickers, years, top_k):
+    if not sem_query.strip():
+        return error_html("Enter a semantic query.")
+    try:
+        resp = run_search_hybrid(sem_query, txt_filter, tickers, years, int(top_k))
+    except Exception as e:
+        return error_html(f"Search failed: {e}")
+    terms = fq.highlight_terms({"scoring": "text", "bm25": txt_filter})
+    return results_html([match_record(m) for m in resp.matches], terms)
+
+
+def builder_state(values: list) -> dict:
+    it = iter(values)
+    scoring, bm25, lucene_mode, raw = next(it), next(it), next(it), next(it)
+    clauses = []
+    for _ in range(N_CLAUSES):
+        occur, kind, value, slop, boost = (next(it) for _ in range(5))
+        if str(value or "").strip():
+            clauses.append({"occur": occur, "kind": kind, "value": value, "slop": slop or 0, "boost": boost or 1.0})
+    text_filters = []
+    for _ in range(N_TEXT_FILTERS):
+        op, value, negate = next(it), next(it), next(it)
+        if str(value or "").strip():
+            text_filters.append({"op": op, "value": value, "negate": bool(negate)})
+    join, tickers, excl, year_from, year_to, chunk_min, chunk_max = (next(it) for _ in range(7))
+
+    year_range = None
+    if year_from != ANY_YEAR or year_to != ANY_YEAR:
+        lo = int(year_from) if year_from != ANY_YEAR else YEARS[0]
+        hi = int(year_to) if year_to != ANY_YEAR else YEARS[-1]
+        year_range = (min(lo, hi), max(lo, hi))
+    chunk_range = None
+    if chunk_min is not None or chunk_max is not None:
+        chunk_range = (int(chunk_min or 0), int(chunk_max if chunk_max is not None else 100000))
+
+    return {
+        "scoring": scoring,
+        "bm25": bm25,
+        "lucene_mode": lucene_mode,
+        "clauses": clauses,
+        "raw": raw,
+        "text_filters": text_filters,
+        "text_filter_join": join,
+        "tickers": tickers or [],
+        "exclude_tickers": excl or [],
+        "year_range": year_range,
+        "chunk_range": chunk_range,
+    }
+
+
+def builder_preview(top_k, *values):
+    try:
+        req = fq.build_request(builder_state(list(values)), int(top_k), INCLUDE_FIELDS)
+    except fq.QueryError as e:
+        return "", "", f"⚠️ {e}"
+    return json.dumps(req, indent=2), fq.to_python(req), ""
+
+
+def builder_values_from_state(state: dict) -> list:
+    values = [state["scoring"], state["bm25"], state["lucene_mode"], state["raw"]]
+    clauses = [c for c in state["clauses"] if str(c.get("value") or "").strip()]
+    for i in range(N_CLAUSES):
+        c = clauses[i] if i < len(clauses) else {}
+        values += [c.get("occur", "SHOULD"), c.get("kind", "term(s)"), c.get("value", ""),
+                   c.get("slop", 0), c.get("boost", 1.0)]
+    for i in range(N_TEXT_FILTERS):
+        f = state["text_filters"][i] if i < len(state["text_filters"]) else {}
+        values += [f.get("op", "$match_phrase"), f.get("value", ""), f.get("negate", False)]
+    yr = state["year_range"]
+    ch = state["chunk_range"]
+    values += [
+        state["text_filter_join"],
+        list(state["tickers"]),
+        list(state["exclude_tickers"]),
+        str(yr[0]) if yr else ANY_YEAR,
+        str(yr[1]) if yr else ANY_YEAR,
+        ch[0] if ch else None,
+        ch[1] if ch else None,
+    ]
+    return values
+
+
+def visibility(scoring: str, lucene_mode: str):
+    lucene = scoring == "query_string"
+    return (
+        gr.update(visible=not lucene),
+        gr.update(visible=lucene),
+        gr.update(visible=lucene and lucene_mode == "clauses"),
+        gr.update(visible=lucene and lucene_mode == "raw"),
+    )
+
+
+def load_builder_example(name: str):
+    ex = BUILDER_EXAMPLES.get(name)
+    state = ex["state"] if ex else fq.BLANK
+    info = f"ℹ️ **Demonstrates:** {ex['shows']}\n\n{ex['description']}" if ex else ""
+    return builder_values_from_state(state) + list(visibility(state["scoring"], state["lucene_mode"])) + [info]
+
+
+def on_lucene_mode(lucene_mode, scoring, raw, *clause_values):
+    vis = visibility(scoring, lucene_mode)
+    if lucene_mode == "raw" and not str(raw or "").strip():
+        rows = [clause_values[i:i + 5] for i in range(0, len(clause_values), 5)]
+        clauses = [{"occur": o, "kind": k, "value": v, "slop": s, "boost": b} for o, k, v, s, b in rows]
+        try:
+            raw = fq.clauses_to_lucene(clauses)
+        except fq.QueryError:
+            pass
+    return (*vis, raw)
+
+
+def do_builder(top_k, *values):
+    state = builder_state(list(values))
+    try:
+        req = fq.build_request(state, int(top_k), INCLUDE_FIELDS)
+        resp = search(req)
+    except fq.QueryError as e:
+        return error_html(str(e))
+    except Exception as e:
+        return error_html(f"Search failed: {e}")
+    return results_html([match_record(m) for m in resp.matches], fq.highlight_terms(state))
+
+
+def compare_example(name: str):
+    ex = next((e for e in fq.MODE_EXAMPLES["compare"] if e["name"] == name), None)
+    n_builder = len(load_builder_example(BLANK))
+    if not ex:
+        return [gr.update(), gr.update(), ""] + [gr.update()] * n_builder
+    builder = load_builder_example(ex["builder_example"]) if ex.get("builder_example") else [gr.update()] * n_builder
+    return [ex["values"]["c_query"], ex["values"]["c_source"], f"ℹ️ {ex['description']}"] + builder
+
+
+def compare_source_preview(source, top_k, *values):
+    if source != "builder":
+        return gr.update(visible=False), gr.update(visible=False)
+    try:
+        req = fq.build_request(builder_state(list(values)), int(top_k), INCLUDE_FIELDS)
+    except fq.QueryError as e:
+        return gr.update(visible=True, value=f"// Query builder: {e}"), gr.update(visible=False)
+    shown = {k: req[k] for k in ("score_by", "filter") if k in req}
+    return gr.update(visible=True, value=json.dumps(shown, indent=2)), gr.update(visible="filter" in req)
+
+
+def do_compare(query, source, share_filters, tickers, years, top_k, *values):
+    empty = ("", "", pd.DataFrame(), "", "")
+    if not str(query or "").strip():
+        return (error_html("Enter a query."),) + empty[1:]
+    dense_extra = None
+    if source == "builder":
+        fts_state = builder_state(list(values))
+        if share_filters:
+            dense_extra = fq.build_filter(fts_state)
+    else:
+        fts_state = {**fq.BLANK, "scoring": "text", "bm25": query}
+    try:
+        res = run_compare(query, fts_state, metadata_filter(tickers, years), dense_extra, int(top_k))
+    except fq.QueryError as e:
+        return (error_html(f"Query builder: {e}"),) + empty[1:]
+    except Exception as e:
+        return (error_html(f"Search failed: {e}"),) + empty[1:]
+
+    stats = compare_stats(res)
+    dense, fts, terms = res["dense"], res["fts"], res["terms"]
+    dense_rank, fts_rank, shared, union = stats["dense_rank"], stats["fts_rank"], stats["shared"], stats["union"]
+
+    metrics = "".join([
+        metric_html("Overlap", f"{len(shared)} / {stats['k']}", f"Jaccard {len(shared) / len(union):.0%}" if union else ""),
+        metric_html("Only dense", str(len(dense_rank.keys() - shared))),
+        metric_html("Only full-text", str(len(fts_rank.keys() - shared))),
+        metric_html("Dense latency", f"{res['embed_ms'] + res['dense_ms']:.0f} ms",
+                    f"embed {res['embed_ms']:.0f} + search {res['dense_ms']:.0f}"),
+        metric_html("Full-text latency", f"{res['fts_ms']:.0f} ms"),
+    ])
+    summary = f"<div class='metrics'>{metrics}</div>"
+    if stats["cov_dense"] is not None:
+        summary += (
+            f"<div class='count'><b>Keyword coverage</b> — average share of the full-text terms "
+            f"({html.escape(', '.join(terms))}) present in each result: dense <b>{stats['cov_dense']:.0%}</b> vs "
+            f"full-text <b>{stats['cov_fts']:.0%}</b>. Low dense coverage means it is finding paraphrases and related "
+            "concepts the keywords miss.</div>"
+        )
+
+    def side(rows, other_ranks, other_name, title):
+        cards = []
+        for i, r in enumerate(rows, 1):
+            hits, n = keyword_coverage(r["text"], terms)
+            cov = f" · keywords {hits}/{n}" if n else ""
+            cards.append(card_html(r, terms, rank=i, badge=rank_badge(r["id"], other_ranks, other_name) + cov))
+        body = "".join(cards) or "<div class='count'>No results.</div>"
+        return f"<div><h4>{title}</h4>{body}</div>"
+
+    side_by_side = (
+        "<div class='cols'>"
+        + side(dense, fts_rank, "full-text", "Dense (semantic)")
+        + side(fts, dense_rank, "dense", "Full-text (BM25 / Lucene)")
+        + "</div>"
+    )
+
+    fused_cards = []
+    for i, (doc_id, score) in enumerate(rrf_fuse(stats), 1):
+        r = {**stats["by_id"][doc_id], "score": score}
+        d, f = dense_rank.get(doc_id), fts_rank.get(doc_id)
+        fused_cards.append(card_html(r, terms, rank=i, badge=f"dense #{d if d else '—'} · full-text #{f if f else '—'}"))
+    fused = (
+        f"<div class='count'>Reciprocal rank fusion merges both lists client-side: score = Σ 1 / ({RRF_K} + rank). "
+        "Documents found by both searches rise to the top — a preview of what a two-query hybrid returns.</div>"
+        + "".join(fused_cards)
+    )
+
+    return summary, side_by_side, pd.DataFrame(rank_rows(res, stats)), fused, json.dumps(res["fts_req"], indent=2)
+
+
+with gr.Blocks(title="SEC FTS Hybrid Comparison") as app:
+    gr.Markdown(
+        "# SEC 10-K Search — Pinecone full-text search vs dense vectors\n"
+        "Six companies (AAPL, AMZN, F, GM, MSFT, ORCL) · 10-K filings 2019–2024 · ~24k chunks. "
+        "Full-text search uses Pinecone's document-schema BM25 / Lucene; dense search uses OpenAI "
+        "`text-embedding-3-small`."
+    )
+    if missing_env():
+        gr.Markdown(
+            f"⚠️ **Missing environment variable(s): {', '.join(missing_env())}.** "
+            "Set them in `.env` locally, or as Secrets in the Space settings."
+        )
+
+    with gr.Sidebar():
+        gr.Markdown("### Filters")
+        side_tickers = gr.CheckboxGroup(TICKERS, label="Ticker", info="Empty = all companies")
+        side_years = gr.CheckboxGroup([str(y) for y in YEARS], label="Year", info="Empty = all years")
+        top_k = gr.Slider(3, 25, value=10, step=1, label="Results")
+        gr.Markdown("_The Query builder tab has its own filters; these apply to the other tabs._")
+
+    with gr.Tabs():
+        with gr.Tab("Full-text"):
+            ft_ex = gr.Dropdown([BLANK] + [e["name"] for e in fq.MODE_EXAMPLES["fulltext"]], value=BLANK, label="Example queries")
+            ft_info = gr.Markdown()
+            ft_query = gr.Textbox(label="Keyword query", placeholder="e.g. revenue growth operating income")
+            ft_btn = gr.Button("Search", variant="primary")
+            ft_out = gr.HTML()
+
+        with gr.Tab("Query builder (full-text)"):
+            b_ex = gr.Dropdown([BLANK] + list(BUILDER_EXAMPLES), value=BLANK, label="Example queries",
+                               info="Load a pre-built complex query into the builder, then tweak it.")
+            b_info = gr.Markdown()
+            with gr.Accordion("Syntax cheat sheet", open=False):
+                gr.Markdown(CHEAT_SHEET)
+
+            gr.Markdown("### 1 · Scoring (ranks results)")
+            b_scoring = gr.Radio([("Lucene (query_string)", "query_string"), ("BM25 keywords (text)", "text")],
+                                 value="query_string", show_label=False)
+            b_bm25 = gr.Textbox(label="BM25 keywords", placeholder="e.g. battery cells charging range", visible=False)
+            b_lucene_mode = gr.Radio([("Clause builder", "clauses"), ("Raw Lucene", "raw")], value="clauses", label="Lucene input")
+            with gr.Group() as b_clause_group:
+                gr.Markdown("Each row is one clause. **MUST** → `+`, **MUST NOT** → `-`, **SHOULD** → optional (adds score). "
+                            "Slop applies to phrases; boost multiplies a clause's weight. Empty rows are ignored.")
+                clause_comps = []
+                for i in range(N_CLAUSES):
+                    with gr.Row(equal_height=True):
+                        clause_comps += [
+                            gr.Dropdown(fq.OCCURS, value="SHOULD", label="Occur", scale=1, min_width=110),
+                            gr.Dropdown(fq.KINDS, value="term(s)", label="Kind", scale=1, min_width=120),
+                            gr.Textbox(label="Value", scale=3, min_width=160),
+                            gr.Number(value=0, label="Slop ~N", precision=0, minimum=0, maximum=50, scale=1, min_width=80),
+                            gr.Number(value=1.0, label="Boost ^N", minimum=0.1, maximum=20, step=0.5, scale=1, min_width=80),
+                        ]
+            b_raw = gr.Textbox(label="Lucene query", lines=3, visible=False,
+                               placeholder='text:(("supply chain" OR semiconductor) AND shortage) NOT text:(covid)')
+
+            gr.Markdown("### 2 · Filters (hard constraints)")
+            gr.Markdown("**Text-match filters** on the `text` field. Empty rows are ignored.")
+            tf_comps = []
+            for i in range(N_TEXT_FILTERS):
+                with gr.Row(equal_height=True):
+                    tf_comps += [
+                        gr.Dropdown(fq.MATCH_OPS, value="$match_phrase", label="Operator", scale=1, min_width=140),
+                        gr.Textbox(label="Value", scale=3, min_width=160),
+                        gr.Checkbox(label="NOT", value=False, scale=0, min_width=70),
+                    ]
+            b_join = gr.Radio([("AND ($and) — every condition", "all"), ("OR ($or) — at least one", "any")],
+                              value="all", label="Combine text-match filters with")
+            gr.Markdown("**Metadata filters**")
+            with gr.Row():
+                b_tickers = gr.CheckboxGroup(TICKERS, label="Ticker is one of ($in)")
+                b_excl = gr.CheckboxGroup(TICKERS, label="Ticker is not ($nin)")
+            with gr.Row():
+                year_choices = [ANY_YEAR] + [str(y) for y in YEARS]
+                b_year_from = gr.Dropdown(year_choices, value=ANY_YEAR, label="Year from ($gte)")
+                b_year_to = gr.Dropdown(year_choices, value=ANY_YEAR, label="Year to ($lte)")
+                b_chunk_min = gr.Number(value=None, label="Chunk index ≥", precision=0, minimum=0)
+                b_chunk_max = gr.Number(value=None, label="Chunk index ≤", precision=0, minimum=0)
+
+            gr.Markdown("### 3 · Request")
+            b_warn = gr.Markdown()
+            with gr.Tabs():
+                with gr.Tab("JSON"):
+                    b_json = gr.Code(language="json", interactive=False, show_label=False)
+                with gr.Tab("Python"):
+                    b_py = gr.Code(language="python", interactive=False, show_label=False)
+            b_btn = gr.Button("Search", variant="primary")
+            b_out = gr.HTML()
+
+        with gr.Tab("Semantic (dense)"):
+            sem_ex = gr.Dropdown([BLANK] + [e["name"] for e in fq.MODE_EXAMPLES["semantic"]], value=BLANK, label="Example queries")
+            sem_info = gr.Markdown()
+            sem_query = gr.Textbox(label="Semantic query", placeholder="e.g. risks related to supply chain disruption")
+            sem_btn = gr.Button("Search", variant="primary")
+            sem_out = gr.HTML()
+
+        with gr.Tab("Hybrid (semantic + text filter)"):
+            hy_ex = gr.Dropdown([BLANK] + [e["name"] for e in fq.MODE_EXAMPLES["hybrid"]], value=BLANK, label="Example queries")
+            hy_info = gr.Markdown()
+            with gr.Row():
+                hy_sem = gr.Textbox(label="Semantic query (drives ranking)", placeholder="e.g. cloud infrastructure investment")
+                hy_txt = gr.Textbox(label="Must-contain keywords (full-text filter)", placeholder="e.g. Azure AWS capital expenditure",
+                                    info="All tokens must appear in the chunk ($match_all).")
+            hy_btn = gr.Button("Search", variant="primary")
+            hy_out = gr.HTML()
+
+        with gr.Tab("Compare: dense vs full-text"):
+            c_ex = gr.Dropdown([BLANK] + [e["name"] for e in fq.MODE_EXAMPLES["compare"]], value=BLANK, label="Example queries")
+            c_info = gr.Markdown()
+            c_query = gr.Textbox(label="Query", placeholder="e.g. risks from rising interest rates on consumer demand",
+                                 info="Embedded for the dense search. Also used as BM25 keywords unless the full-text side uses the query builder.")
+            c_source = gr.Radio([("Same query as BM25 keywords", "same"), ("Current query-builder query", "builder")],
+                                value="same", label="Full-text side")
+            c_builder_preview = gr.Code(language="json", interactive=False, label="Query-builder request (full-text side)", visible=False)
+            c_share = gr.Checkbox(label="Apply the builder's filters to the dense search too", value=False, visible=False,
+                                  info="Makes the dense side a hybrid, isolating the effect of the ranking signal.")
+            gr.Markdown("_Sidebar ticker / year filters apply to both searches._")
+            c_btn = gr.Button("Compare", variant="primary")
+            c_summary = gr.HTML()
+            with gr.Tabs():
+                with gr.Tab("Side by side"):
+                    c_side = gr.HTML()
+                with gr.Tab("Rank comparison"):
+                    c_table = gr.Dataframe(interactive=False, wrap=True)
+                    gr.Markdown("_Δ rank < 0 → dense ranks it higher; > 0 → full-text ranks it higher. Blank rank = not in that top-k._")
+                with gr.Tab("Fused (RRF)"):
+                    c_rrf = gr.HTML()
+            with gr.Accordion("Full-text request sent", open=False):
+                c_req = gr.Code(language="json", interactive=False, show_label=False)
+
+    side_inputs = [side_tickers, side_years, top_k]
+    builder_inputs = [b_scoring, b_bm25, b_lucene_mode, b_raw, *clause_comps, *tf_comps,
+                      b_join, b_tickers, b_excl, b_year_from, b_year_to, b_chunk_min, b_chunk_max]
+    builder_vis = [b_bm25, b_lucene_mode, b_clause_group, b_raw]
+
+    ft_ex.change(lambda n: mode_example("fulltext", n, ["ft_query"]), ft_ex, [ft_query, ft_info])
+    sem_ex.change(lambda n: mode_example("semantic", n, ["sem_query"]), sem_ex, [sem_query, sem_info])
+    hy_ex.change(lambda n: mode_example("hybrid", n, ["hy_sem", "hy_txt"]), hy_ex, [hy_sem, hy_txt, hy_info])
+
+    gr.on([ft_btn.click, ft_query.submit], do_fulltext, [ft_query, *side_inputs], ft_out)
+    gr.on([sem_btn.click, sem_query.submit], do_semantic, [sem_query, *side_inputs], sem_out)
+    gr.on([hy_btn.click, hy_sem.submit, hy_txt.submit], do_hybrid, [hy_sem, hy_txt, *side_inputs], hy_out)
+
+    b_ex.change(load_builder_example, b_ex, builder_inputs + builder_vis + [b_info])
+    b_scoring.change(visibility, [b_scoring, b_lucene_mode], builder_vis)
+    b_lucene_mode.change(on_lucene_mode, [b_lucene_mode, b_scoring, b_raw, *clause_comps], builder_vis + [b_raw])
+    gr.on([c.change for c in builder_inputs] + [top_k.change, app.load], builder_preview,
+          [top_k, *builder_inputs], [b_json, b_py, b_warn])
+    b_btn.click(do_builder, [top_k, *builder_inputs], b_out)
+
+    c_ex.change(compare_example, c_ex, [c_query, c_source, c_info] + builder_inputs + builder_vis + [b_info])
+    gr.on([c_source.change] + [c.change for c in builder_inputs], compare_source_preview,
+          [c_source, top_k, *builder_inputs], [c_builder_preview, c_share])
+    gr.on([c_btn.click, c_query.submit], do_compare, [c_query, c_source, c_share, *side_inputs, *builder_inputs],
+          [c_summary, c_side, c_table, c_rrf, c_req])
+
+demo = app
+
+if __name__ == "__main__":
+    app.launch(server_name="0.0.0.0", server_port=7860, ssr_mode=False, css=CSS, theme=gr.themes.Soft())

fts_queries.py

@@ -0,0 +1,386 @@
+import json
+import re
+
+TEXT_FIELD = "text"
+
+OCCURS = ["SHOULD", "MUST", "MUST NOT"]
+OCCUR_PREFIX = {"SHOULD": "", "MUST": "+", "MUST NOT": "-"}
+KINDS = ["term(s)", "phrase", "phrase prefix"]
+MATCH_OPS = ["$match_phrase", "$match_all", "$match_any"]
+
+LUCENE_KEYWORDS = {"and", "or", "not", "text"}
+STOP_WORDS = {
+    "a", "an", "the", "of", "on", "in", "to", "for", "from", "by", "with", "at", "as", "is", "are",
+    "was", "be", "it", "its", "this", "that", "these", "those", "our", "we", "their", "about", "into",
+}
+
+
+class QueryError(ValueError):
+    pass
+
+
+def clause_to_lucene(c: dict) -> str | None:
+    value = str(c.get("value") or "").strip().replace('"', "")
+    if not value:
+        return None
+    kind = c.get("kind") or "term(s)"
+    slop = int(c.get("slop") or 0)
+    boost = float(c.get("boost") or 1.0)
+
+    if kind == "phrase":
+        expr = f'"{value}"' + (f"~{slop}" if slop > 0 else "")
+    elif kind == "phrase prefix":
+        if len(value.split()) < 2:
+            raise QueryError(
+                f"Phrase prefix '{value}' needs at least two words "
+                '(e.g. "artificial intel") — single-term wildcards are not supported.'
+            )
+        expr = f'"{value}"*'
+    else:
+        expr = value
+
+    prefix = OCCUR_PREFIX.get(c.get("occur") or "SHOULD", "")
+    if kind == "term(s)" and len(value.split()) > 1 and (prefix or boost != 1.0):
+        expr = f"({value})"
+    if boost != 1.0:
+        expr += f"^{boost:g}"
+    return prefix + expr
+
+
+def clauses_to_lucene(clauses: list[dict]) -> str:
+    parts, positive = [], 0
+    for c in clauses:
+        expr = clause_to_lucene(c)
+        if expr is None:
+            continue
+        parts.append(expr)
+        if c.get("occur") != "MUST NOT":
+            positive += 1
+    if not parts:
+        return ""
+    if positive == 0:
+        raise QueryError("Add at least one SHOULD or MUST clause — a query of only exclusions is rejected.")
+    return f"{TEXT_FIELD}:({' '.join(parts)})"
+
+
+def build_filter(state: dict) -> dict | None:
+    conds = []
+
+    text_conds = []
+    for f in state.get("text_filters") or []:
+        value = str(f.get("value") or "").strip()
+        op = f.get("op")
+        if not value or op not in MATCH_OPS:
+            continue
+        cond = {TEXT_FIELD: {op: value}}
+        text_conds.append({"$not": cond} if f.get("negate") else cond)
+    if len(text_conds) > 1 and state.get("text_filter_join") == "any":
+        conds.append({"$or": text_conds})
+    else:
+        conds.extend(text_conds)
+
+    if state.get("tickers"):
+        conds.append({"ticker": {"$in": list(state["tickers"])}})
+    if state.get("exclude_tickers"):
+        conds.append({"ticker": {"$nin": list(state["exclude_tickers"])}})
+    if state.get("year_range"):
+        lo, hi = state["year_range"]
+        conds.append({"year": {"$gte": lo, "$lte": hi}})
+    if state.get("chunk_range"):
+        lo, hi = state["chunk_range"]
+        conds.append({"chunk_index": {"$gte": lo, "$lte": hi}})
+
+    if not conds:
+        return None
+    return conds[0] if len(conds) == 1 else {"$and": conds}
+
+
+def build_score_by(state: dict) -> list[dict]:
+    if state["scoring"] == "text":
+        query = (state.get("bm25") or "").strip()
+        if not query:
+            raise QueryError("Enter BM25 keywords.")
+        return [{"type": "text", "field": TEXT_FIELD, "query": query}]
+
+    if state.get("lucene_mode") == "raw":
+        query = (state.get("raw") or "").strip()
+    else:
+        query = clauses_to_lucene(state.get("clauses") or [])
+    if not query:
+        raise QueryError("The Lucene query is empty.")
+    return [{"type": "query_string", "query": query}]
+
+
+def build_request(state: dict, top_k: int, include_fields: list[str]) -> dict:
+    req = {
+        "namespace": "__default__",
+        "top_k": top_k,
+        "score_by": build_score_by(state),
+        "include_fields": include_fields,
+    }
+    filt = build_filter(state)
+    if filt:
+        req["filter"] = filt
+    return req
+
+
+def to_python(req: dict) -> str:
+    lines = ["resp = idx.documents.search("]
+    for key, value in req.items():
+        rendered = json.dumps(value, indent=4).replace("\n", "\n    ")
+        lines.append(f"    {key}={rendered},")
+    lines.append(")")
+    return "\n".join(lines)
+
+
+def highlight_terms(state: dict) -> list[str]:
+    sources = []
+    if state["scoring"] == "text":
+        sources.append(state.get("bm25") or "")
+    elif state.get("lucene_mode") == "raw":
+        raw = state.get("raw") or ""
+        raw = re.sub(r"-\"[^\"]*\"|-\w+|NOT\s+\w+:\([^)]*\)|NOT\s+\"[^\"]*\"|NOT\s+\w+", " ", raw)
+        sources.append(raw)
+    else:
+        sources += [str(c.get("value") or "") for c in state.get("clauses") or [] if c.get("occur") != "MUST NOT"]
+    sources += [str(f.get("value") or "") for f in state.get("text_filters") or [] if not f.get("negate")]
+
+    terms = set()
+    for s in sources:
+        for word in re.findall(r"[A-Za-z][A-Za-z0-9']+", s):
+            if word.lower() not in LUCENE_KEYWORDS | STOP_WORDS:
+                terms.add(word.lower())
+    return sorted(terms, key=len, reverse=True)
+
+
+def _clause(value, occur="SHOULD", kind="term(s)", slop=0, boost=1.0):
+    return {"occur": occur, "kind": kind, "value": value, "slop": slop, "boost": boost}
+
+
+def _tf(op, value, negate=False):
+    return {"op": op, "value": value, "negate": negate}
+
+
+BLANK = {
+    "scoring": "query_string",
+    "bm25": "",
+    "lucene_mode": "clauses",
+    "clauses": [_clause("")],
+    "raw": "",
+    "text_filters": [],
+    "text_filter_join": "all",
+    "tickers": [],
+    "exclude_tickers": [],
+    "year_range": None,
+    "chunk_range": None,
+}
+
+
+def _example(**overrides) -> dict:
+    return {**BLANK, **overrides}
+
+
+EXAMPLES = [
+    {
+        "name": "Supply-chain shortages, excluding COVID",
+        "shows": "Boolean nesting — AND / OR / NOT with a phrase inside a group",
+        "description": "Chunks that talk about a shortage of either the supply chain or semiconductors, "
+        "but never mention COVID. Parentheses group the OR before the AND.",
+        "state": _example(
+            lucene_mode="raw",
+            raw='text:(("supply chain" OR semiconductor) AND shortage) NOT text:(covid)',
+        ),
+    },
+    {
+        "name": "Cyber incidents (required / excluded / boosted)",
+        "shows": "Clause builder — MUST (+), MUST NOT (−), SHOULD with a boost",
+        "description": "Every result must mention cybersecurity; ransomware mentions count triple, "
+        "breach is a nice-to-have, and chunks about insurance are dropped.",
+        "state": _example(
+            clauses=[
+                _clause("cybersecurity", occur="MUST"),
+                _clause("ransomware", boost=3.0),
+                _clause("breach"),
+                _clause("insurance", occur="MUST NOT"),
+            ],
+        ),
+    },
+    {
+        "name": "Rising interest rates (proximity)",
+        "shows": 'Phrase slop — "interest rates increase"~5',
+        "description": "The three words must appear within 5 positions of each other, so "
+        '"increases in interest rates" and "interest rates may continue to increase" both match. '
+        "Stemming lets increase match increases/increased.",
+        "state": _example(
+            clauses=[_clause("interest rates increase", kind="phrase", slop=5)],
+        ),
+    },
+    {
+        "name": "China trade & tariffs (term boost)",
+        "shows": "Term boosting — tariffs^3 outweighs trade and china",
+        "description": "BM25 over three terms, but tariffs is weighted 3×. Compare with the boost "
+        "removed to see ranking shift toward generic trade/China text.",
+        "state": _example(
+            clauses=[
+                _clause("tariffs", boost=3.0),
+                _clause("trade"),
+                _clause("china"),
+            ],
+        ),
+    },
+    {
+        "name": "AI mentions (phrase prefix)",
+        "shows": 'Phrase prefix — "artificial intel"* matches intelligence / intelligent',
+        "description": "The last word is matched as a prefix. Combined with a required cloud term "
+        "and restricted to 2022+ filings. Note: prefix matches all share a constant score.",
+        "state": _example(
+            clauses=[
+                _clause("artificial intel", occur="MUST", kind="phrase prefix"),
+                _clause("cloud", occur="MUST"),
+            ],
+            year_range=(2022, 2024),
+        ),
+    },
+    {
+        "name": "EV batteries at Ford & GM, 2022+",
+        "shows": "BM25 ranking + $match_phrase hard filter + metadata $in / range",
+        "description": 'Plain BM25 over battery terms, restricted to chunks containing the exact phrase '
+        '"electric vehicles", from Ford or GM, filed 2022–2024.',
+        "state": _example(
+            scoring="text",
+            bm25="battery cells charging range",
+            text_filters=[_tf("$match_phrase", "electric vehicles")],
+            tickers=["f", "gm"],
+            year_range=(2022, 2024),
+        ),
+    },
+    {
+        "name": "Cloud growth, no pandemic talk, tech only",
+        "shows": "$not + $match_any exclusion filter, ticker $nin",
+        "description": "Ranks on cloud revenue growth but excludes any chunk mentioning COVID or "
+        "pandemic, and excludes the automakers entirely.",
+        "state": _example(
+            clauses=[_clause("cloud revenue growth")],
+            text_filters=[_tf("$match_any", "covid pandemic", negate=True)],
+            exclude_tickers=["f", "gm"],
+        ),
+    },
+    {
+        "name": "Regulators: EC or DOJ",
+        "shows": "$or across two $match_phrase filters",
+        "description": "Antitrust / litigation language, limited to chunks that name either the "
+        "European Commission or the Department of Justice.",
+        "state": _example(
+            clauses=[_clause("antitrust", boost=2.0), _clause("litigation"), _clause("investigation")],
+            text_filters=[
+                _tf("$match_phrase", "european commission"),
+                _tf("$match_phrase", "department of justice"),
+            ],
+            text_filter_join="any",
+        ),
+    },
+    {
+        "name": "Inflation & input costs, 2021–2023",
+        "shows": "$match_all (all tokens, any order) + year range",
+        "description": "Ranks on inflation; every result must contain raw, materials and costs somewhere "
+        "in the chunk (any order), from the 2021–2023 filings.",
+        "state": _example(
+            scoring="text",
+            bm25="inflation inflationary pressures",
+            text_filters=[_tf("$match_all", "raw materials costs")],
+            year_range=(2021, 2023),
+        ),
+    },
+    {
+        "name": "Buybacks vs. dividends (required groups)",
+        "shows": "Required OR-groups — +(a OR b) +(c OR d)",
+        "description": "Each result must contain a buyback term AND a dividend term; the phrase "
+        '"share repurchase" is boosted so it outranks bare "repurchase".',
+        "state": _example(
+            lucene_mode="raw",
+            raw='text:(+("share repurchase"^2 OR buyback OR repurchase) +(dividend OR dividends))',
+        ),
+    },
+]
+
+
+MODE_EXAMPLES = {
+    "fulltext": [
+        {
+            "name": "Buybacks & dividends",
+            "description": "BM25 token-OR: chunks with more (and rarer) of these terms rank higher.",
+            "values": {"ft_query": "share repurchase program dividends"},
+        },
+        {
+            "name": "Chip shortage hits production",
+            "description": "Surfaces the automakers' 2021–2022 semiconductor supply disclosures.",
+            "values": {"ft_query": "semiconductor shortage production"},
+        },
+        {
+            "name": "FX risk",
+            "description": "Market-risk language about foreign currency exposure.",
+            "values": {"ft_query": "foreign currency exchange rate risk"},
+        },
+    ],
+    "semantic": [
+        {
+            "name": "Interest-rate exposure (question)",
+            "description": "A natural-language question — dense search matches meaning, not exact words.",
+            "values": {"sem_query": "How is the company exposed to rising interest rates?"},
+        },
+        {
+            "name": "Supplier concentration in Asia",
+            "description": "Filings rarely use this wording; embeddings find single-source and "
+            "outsourcing-partner risk disclosures anyway.",
+            "values": {"sem_query": "Concerns about dependence on a small number of suppliers in Asia"},
+        },
+        {
+            "name": "AI & data-center buildout",
+            "description": "Conceptual query that spans capex, infrastructure and AI discussion.",
+            "values": {"sem_query": "Investments in artificial intelligence and data center capacity"},
+        },
+    ],
+    "hybrid": [
+        {
+            "name": "Cloud growth that names Azure",
+            "description": "Semantic ranking on cloud growth, but only chunks containing \"Azure\".",
+            "values": {"hy_sem": "growth of cloud infrastructure services", "hy_txt": "Azure"},
+        },
+        {
+            "name": "EV transition that names Ultium",
+            "description": "GM's battery platform as a hard keyword; ranking by the EV-transition concept.",
+            "values": {"hy_sem": "transition to electric vehicles and battery supply", "hy_txt": "Ultium"},
+        },
+        {
+            "name": "Competition probes by the EC",
+            "description": "Regulatory-investigation meaning, restricted to chunks mentioning European Commission.",
+            "values": {
+                "hy_sem": "regulatory investigations into competition practices",
+                "hy_txt": "European Commission",
+            },
+        },
+    ],
+    "compare": [
+        {
+            "name": "Interest rates vs consumer demand",
+            "description": "Same words on both sides: BM25 rewards literal term hits, dense rewards the idea.",
+            "values": {"c_query": "risks from rising interest rates on consumer demand", "c_source": "same"},
+        },
+        {
+            "name": "Paraphrase: overseas manufacturing partners",
+            "description": "Wording filings don't use verbatim — expect low overlap and low keyword "
+            "coverage on the dense side, where it still finds outsourcing-risk passages.",
+            "values": {
+                "c_query": "the company depends on a few manufacturing partners overseas",
+                "c_source": "same",
+            },
+        },
+        {
+            "name": "Lucene builder query vs dense",
+            "description": "Loads the \"Supply-chain shortages, excluding COVID\" builder query as the "
+            "full-text side against a dense search for the same topic.",
+            "values": {"c_query": "semiconductor chip shortage impact on production", "c_source": "builder"},
+            "builder_example": "Supply-chain shortages, excluding COVID",
+        },
+    ],
+}

requirements.txt

@@ -0,0 +1,5 @@
+pinecone==10.0.0
+openai==3.22.1
+pandas==3.0.6
+python-dotenv==1.2.3
+spaces==0.51.3

search_core.py

@@ -0,0 +1,237 @@
+import html
+import os
+import re
+import time
+from concurrent.futures import ThreadPoolExecutor
+from functools import lru_cache
+
+from dotenv import load_dotenv
+from openai import OpenAI
+from pinecone import Pinecone
+
+import fts_queries as fq
+
+load_dotenv()
+
+INDEX_NAME = "sec-fts"
+NAMESPACE = "__default__"
+EMBED_MODEL = "text-embedding-3-small"
+
+TICKERS = ["aapl", "amzn", "f", "gm", "msft", "orcl"]
+YEARS = list(range(2019, 2025))
+INCLUDE_FIELDS = ["text", "ticker", "filing_type", "year", "chunk_index"]
+RRF_K = 60
+
+
+def missing_env() -> list[str]:
+    return [k for k in ("PINECONE_API_KEY", "OPENAI_API_KEY") if not os.environ.get(k)]
+
+
+@lru_cache(maxsize=1)
+def get_index():
+    return Pinecone(api_key=os.environ["PINECONE_API_KEY"]).index(name=INDEX_NAME)
+
+
+@lru_cache(maxsize=1)
+def get_openai() -> OpenAI:
+    return OpenAI(api_key=os.environ["OPENAI_API_KEY"])
+
+
+@lru_cache(maxsize=512)
+def _embed(text: str) -> tuple[float, ...]:
+    resp = get_openai().embeddings.create(model=EMBED_MODEL, input=[text])
+    return tuple(resp.data[0].embedding)
+
+
+def embed(text: str) -> list[float]:
+    return list(_embed(text))
+
+
+def metadata_filter(tickers: list[str], years: list[int]) -> dict | None:
+    filt = {}
+    if tickers:
+        filt["ticker"] = {"$in": list(tickers)}
+    if years:
+        filt["year"] = {"$in": [int(y) for y in years]}
+    return filt or None
+
+
+def and_filters(*filters: dict | None) -> dict | None:
+    parts = [f for f in filters if f]
+    if not parts:
+        return None
+    return parts[0] if len(parts) == 1 else {"$and": parts}
+
+
+def search(req: dict):
+    return get_index().documents.search(**req)
+
+
+def dense_request(query: str, top_k: int, filt: dict | None = None) -> dict:
+    req = {
+        "namespace": NAMESPACE,
+        "top_k": top_k,
+        "score_by": [{"type": "dense_vector", "field": "embedding", "values": embed(query)}],
+        "include_fields": INCLUDE_FIELDS,
+    }
+    if filt:
+        req["filter"] = filt
+    return req
+
+
+def run_search_text(query: str, tickers: list, years: list, top_k: int):
+    req = {
+        "namespace": NAMESPACE,
+        "top_k": top_k,
+        "score_by": [{"type": "text", "field": "text", "query": query}],
+        "include_fields": INCLUDE_FIELDS,
+    }
+    filt = metadata_filter(tickers, years)
+    if filt:
+        req["filter"] = filt
+    return search(req)
+
+
+def run_search_semantic(query: str, tickers: list, years: list, top_k: int):
+    return search(dense_request(query, top_k, metadata_filter(tickers, years)))
+
+
+def run_search_hybrid(semantic_query: str, text_filter: str, tickers: list, years: list, top_k: int):
+    filt = metadata_filter(tickers, years) or {}
+    if text_filter.strip():
+        filt["text"] = {"$match_all": text_filter.strip()}
+    return search(dense_request(semantic_query, top_k, filt or None))
+
+
+def term_pattern(terms: list[str]) -> re.Pattern | None:
+    if not terms:
+        return None
+    stems = [re.escape(t[:-2] if len(t) > 5 else t) for t in terms]
+    return re.compile(r"\b(" + "|".join(stems) + r")\w*", re.IGNORECASE)
+
+
+def highlight(text: str, terms: list[str]) -> tuple[str, str]:
+    escaped = html.escape(text)
+    pattern = term_pattern(terms)
+    if not pattern:
+        return escaped[:300] + ("..." if len(escaped) > 300 else ""), escaped
+    marked = pattern.sub(lambda m: f"<mark>{m.group(0)}</mark>", escaped)
+
+    first = pattern.search(escaped)
+    start = max(0, first.start() - 150) if first else 0
+    snippet_raw = escaped[start : start + 450]
+    snippet = pattern.sub(lambda m: f"<mark>{m.group(0)}</mark>", snippet_raw)
+    snippet = ("..." if start else "") + snippet + ("..." if start + 450 < len(escaped) else "")
+    return snippet, marked
+
+
+def keyword_coverage(text: str, terms: list[str]) -> tuple[int, int]:
+    hits = sum(1 for t in terms if term_pattern([t]).search(text))
+    return hits, len(terms)
+
+
+def match_record(m) -> dict:
+    d = m.to_dict()
+    return {
+        "id": getattr(m, "_id", d.get("_id")),
+        "score": getattr(m, "_score", getattr(m, "score", d.get("_score", 0.0))),
+        "ticker": str(d.get("ticker", "?")).upper(),
+        "year": int(d.get("year", 0)),
+        "filing": str(d.get("filing_type", "")).upper(),
+        "chunk": int(d["chunk_index"]) if d.get("chunk_index") is not None else "?",
+        "text": str(d.get("text", "")),
+    }
+
+
+def _timed(fn):
+    start = time.perf_counter()
+    result = fn()
+    return result, (time.perf_counter() - start) * 1000
+
+
+def run_compare(query: str, fts_state: dict, base_filter: dict | None, dense_extra_filter: dict | None, top_k: int) -> dict:
+    fts_req = fq.build_request(fts_state, top_k, INCLUDE_FIELDS)
+    fts_req["filter"] = and_filters(fts_req.get("filter"), base_filter)
+    if not fts_req["filter"]:
+        del fts_req["filter"]
+
+    def dense():
+        _, embed_ms = _timed(lambda: embed(query))
+        resp, search_ms = _timed(lambda: search(dense_request(query, top_k, and_filters(base_filter, dense_extra_filter))))
+        return resp, embed_ms, search_ms
+
+    with ThreadPoolExecutor(max_workers=2) as pool:
+        dense_future = pool.submit(dense)
+        fts_future = pool.submit(_timed, lambda: search(fts_req))
+        dense_resp, embed_ms, dense_ms = dense_future.result()
+        fts_resp, fts_ms = fts_future.result()
+
+    return {
+        "dense": [match_record(m) for m in dense_resp.matches],
+        "fts": [match_record(m) for m in fts_resp.matches],
+        "embed_ms": embed_ms,
+        "dense_ms": dense_ms,
+        "fts_ms": fts_ms,
+        "fts_req": fts_req,
+        "terms": fq.highlight_terms(fts_state),
+    }
+
+
+def compare_stats(res: dict) -> dict:
+    dense, fts, terms = res["dense"], res["fts"], res["terms"]
+    dense_rank = {r["id"]: i for i, r in enumerate(dense, 1)}
+    fts_rank = {r["id"]: i for i, r in enumerate(fts, 1)}
+    shared = dense_rank.keys() & fts_rank.keys()
+    union = dense_rank.keys() | fts_rank.keys()
+
+    def avg_coverage(rows):
+        if not rows or not terms:
+            return None
+        return sum(keyword_coverage(r["text"], terms)[0] / len(terms) for r in rows) / len(rows)
+
+    return {
+        "dense_rank": dense_rank,
+        "fts_rank": fts_rank,
+        "shared": shared,
+        "union": union,
+        "by_id": {r["id"]: r for r in dense + fts},
+        "cov_dense": avg_coverage(dense),
+        "cov_fts": avg_coverage(fts),
+        "k": max(len(dense), len(fts)),
+    }
+
+
+def rank_rows(res: dict, stats: dict) -> list[dict]:
+    terms = res["terms"]
+    rows = []
+    for doc_id in stats["union"]:
+        r = stats["by_id"][doc_id]
+        d, f = stats["dense_rank"].get(doc_id), stats["fts_rank"].get(doc_id)
+        hits, n = keyword_coverage(r["text"], terms)
+        rows.append({
+            "ticker": r["ticker"],
+            "year": r["year"],
+            "chunk": r["chunk"],
+            "dense rank": d,
+            "full-text rank": f,
+            "Δ rank (dense − FTS)": d - f if d and f else None,
+            "found by": "both" if d and f else ("dense" if d else "full-text"),
+            "keywords": f"{hits}/{n}" if n else "",
+            "snippet": r["text"][:140].replace("\n", " "),
+        })
+    rows.sort(key=lambda x: min(x["dense rank"] or 999, x["full-text rank"] or 999))
+    return rows
+
+
+def rrf_fuse(stats: dict) -> list[tuple[str, float]]:
+    fused = {}
+    for ranks in (stats["dense_rank"], stats["fts_rank"]):
+        for doc_id, rank in ranks.items():
+            fused[doc_id] = fused.get(doc_id, 0.0) + 1 / (RRF_K + rank)
+    return sorted(fused.items(), key=lambda kv: kv[1], reverse=True)[: stats["k"]]
+
+
+def rank_badge(doc_id: str, other_ranks: dict, other_name: str) -> str:
+    if doc_id in other_ranks:
+        return f"🔁 also #{other_ranks[doc_id]} in {other_name}"
+    return f"◆ only in {'dense' if other_name == 'full-text' else 'full-text'}"