Add hybrid RAG: XBRL structured-financials layer for numeric questions

README.md

@@ -28,6 +28,12 @@ citations** — instead of making things up.
 
 - **Grounded answers with citations** — every response is backed by excerpts
   from real 10-K filings, shown in an expandable *Sources* panel.
+- **Answers financial figures (hybrid RAG)** — plain text RAG can't read numbers
+  out of financial-statement tables. FinChat also extracts each filing's
+  **XBRL** structured financials (revenue, net income, assets, cash flow…) and
+  a hybrid retriever *guarantees* those facts are in context for numeric
+  questions — so *"What was Apple's FY2023 revenue?"* returns **$383.29 billion**,
+  not a shrug.
 - **Query routing ("knows where to look")** — FinChat detects which company a
   question is about and searches *only* that company's filings via metadata
   filtering, with graceful semantic fallback when the company is ambiguous.
@@ -41,13 +47,14 @@ citations** — instead of making things up.
 ## 🏗️ Architecture
 
 ```
-INGESTION (once)
-  fetch recent 10-Ks from SEC EDGAR ──► split into chunks
-        ──► embed (local model) ──► store vectors + metadata in ChromaDB
+INGESTION (once) — two tracks per filing
+  10-K TEXT       ──► split into chunks ───────────┐
+  XBRL FINANCIALS ──► "label: value" fact chunks ──┴─► embed ──► ChromaDB
 
 QUERY (per question)
-  question ──► detect company ──► embed ──► search (filtered by company)
-        ──► top-k chunks ──► LLM ──► grounded answer + citations
+  question ──► detect company ──► HYBRID retrieve
+        (semantic text chunks + guaranteed XBRL statements for numeric Qs)
+        ──► LLM ──► grounded answer + citations
 ```
 
 | Layer         | Tool                                              |
@@ -57,7 +64,7 @@ QUERY (per question)
 | Vector store  | ChromaDB (persisted locally)                      |
 | LLM           | Llama 3.3 70B via Groq (free)                     |
 | UI            | Streamlit                                         |
-| Data          | Recent SEC 10-K filings via `edgartools` (EDGAR)  |
+| Data          | SEC 10-K text **+ XBRL financials** via `edgartools` |
 | Evaluation    | Capability gold set + FinanceBench, LLM-as-judge  |
 
 **Corpus — 25 recognizable companies (FY2021–2024 10-Ks):** Apple, Microsoft,
@@ -140,8 +147,9 @@ committed index.
 
 ## ⚠️ Limitations
 
-- **Numeric/analytical** questions (ratios, margins) require computation over
-  financial-statement tables — the main gap (see Evaluation).
+- **Direct financial figures** (revenue, net income, assets, cash flow) are now
+  answered from XBRL data. **Computed metrics** (ratios, margin trends,
+  EBITDA-less-capex) still need a calculation layer — the next enhancement.
 - The corpus is scoped to 25 companies' recent 10-Ks to stay laptop-friendly.
 - Not financial advice — a portfolio/educational project.
 

src/financials.py

@@ -0,0 +1,114 @@
+"""Extract structured financial figures from a 10-K's XBRL data.
+
+SEC filings carry machine-readable XBRL financials. We pull the income
+statement, balance sheet, and cash-flow statement as clean "label: value"
+facts and index them alongside the filing text, so FinChat can answer numeric
+questions (revenue, net income, assets, ...) that plain text RAG cannot -- the
+tables in the filing text collapse into unusable "number soup".
+"""
+from __future__ import annotations
+
+import re
+
+import pandas as pd
+from langchain_core.documents import Document
+
+_STATEMENTS = [
+    ("income_statement", "Income Statement"),
+    ("balance_sheet", "Balance Sheet"),
+    ("cash_flow_statement", "Cash Flow Statement"),
+]
+# Synonyms in the chunk header so numeric queries retrieve these chunks.
+_KEYWORDS = ("revenue, sales, income, earnings, profit, margin, assets, "
+            "liabilities, equity, cash flow, expenses, EPS")
+
+
+def _fmt(value, label: str = "") -> str | None:
+    try:
+        v = float(value)
+    except (TypeError, ValueError):
+        return None
+    if pd.isna(v):
+        return None
+    a = abs(v)
+    if "shares" in label.lower() and "per share" not in label.lower():
+        if a >= 1e9:
+            return f"{v / 1e9:,.2f} billion shares"
+        if a >= 1e6:
+            return f"{v / 1e6:,.1f} million shares"
+        return f"{v:,.0f} shares"
+    if a >= 1e9:
+        return f"${v / 1e9:,.2f} billion"
+    if a >= 1e6:
+        return f"${v / 1e6:,.1f} million"
+    if a >= 1000:
+        return f"${v:,.0f}"
+    return f"{v:,.2f}"
+
+
+def _value_column(df: pd.DataFrame, fiscal_year: int):
+    """The statement column for the filing's primary fiscal year."""
+    date_cols = [c for c in df.columns if re.match(r"\d{4}-\d{2}-\d{2}", str(c))]
+    for c in date_cols:
+        if str(c).startswith(str(fiscal_year)):
+            return c
+    return date_cols[0] if date_cols else None
+
+
+def _statement_lines(stmt, fiscal_year: int) -> list[str]:
+    try:
+        df = stmt.to_dataframe()
+    except Exception:
+        return []
+    col = _value_column(df, fiscal_year)
+    if col is None:
+        return []
+    lines = []
+    for _, row in df.iterrows():
+        # skip section headers, per-dimension breakdowns, sub-members
+        if row.get("abstract") or row.get("is_breakdown") or row.get("dimension"):
+            continue
+        label = str(row.get("label") or "").strip()
+        if not label:
+            continue
+        value = _fmt(row.get(col), label)
+        if value is None:
+            continue
+        lines.append(f"{label}: {value}")
+    return lines
+
+
+def financial_documents(filing, ticker: str, company: str,
+                        fiscal_year: int) -> list[Document]:
+    """One Document per financial statement, built from the filing's XBRL data."""
+    try:
+        tenk = filing.obj()
+    except Exception:
+        return []
+
+    docs: list[Document] = []
+    for attr, name in _STATEMENTS:
+        stmt = getattr(tenk, attr, None)
+        if stmt is None:
+            continue
+        lines = _statement_lines(stmt, fiscal_year)
+        if not lines:
+            continue
+        header = (
+            f"{company} ({ticker}) FY{fiscal_year} {name} "
+            f"(financial figures: {_KEYWORDS}; from SEC XBRL data):"
+        )
+        docs.append(
+            Document(
+                page_content=header + "\n" + "\n".join(lines),
+                metadata={
+                    "ticker": ticker,
+                    "company": company,
+                    "year": str(fiscal_year),
+                    "accession": filing.accession_no,
+                    "source": f"{company} 10-K (FY{fiscal_year}) - {name} (XBRL)",
+                    "type": "financials",
+                },
+            )
+        )
+    return docs

src/ingest.py

@@ -24,6 +24,7 @@ from langchain_huggingface import HuggingFaceEmbeddings
 from langchain_chroma import Chroma
 
 from src import config
+from src.financials import financial_documents
 
 # Windows consoles default to cp1252 and crash when print() emits Unicode
 # (arrows, em-dashes, curly quotes from filings). Force UTF-8 output.
@@ -33,8 +34,8 @@ except Exception:
     pass
 
 
-def fetch_filing(ticker: str, fiscal_year: int) -> tuple[str, str] | None:
-    """Return (text, accession_no) for the 10-K whose fiscal period matches.
+def get_filing(ticker: str, fiscal_year: int):
+    """Return the 10-K filing whose fiscal period matches fiscal_year, else None.
 
     Matches on the filing's period_of_report year, so an offset fiscal year
     (e.g. Amcor's June close) still resolves to the right filing.
@@ -42,12 +43,12 @@ def fetch_filing(ticker: str, fiscal_year: int) -> tuple[str, str] | None:
     for f in Company(ticker).get_filings(form="10-K"):
         period = getattr(f, "period_of_report", None)
         if period and str(period)[:4] == str(fiscal_year):
-            return f.text(), f.accession_no
+            return f
     return None
 
 
 def fetch_documents() -> list[Document]:
-    """Fetch every target filing and split it into chunked LangChain Documents."""
+    """Fetch every target filing -> text chunks + structured financial facts."""
     set_identity(config.EDGAR_IDENTITY)
     splitter = RecursiveCharacterTextSplitter(
         chunk_size=config.CHUNK_SIZE,
@@ -57,11 +58,13 @@ def fetch_documents() -> list[Document]:
     docs: list[Document] = []
     for ticker, name, year in config.TARGET_FILINGS:
         print(f"Fetching {ticker} FY{year} 10-K ...", end=" ", flush=True)
-        result = fetch_filing(ticker, year)
-        if result is None:
+        filing = get_filing(ticker, year)
+        if filing is None:
             print("NOT FOUND -- skipping")
             continue
-        text, accession = result
+
+        # 1) Filing TEXT -> chunks (for qualitative questions).
+        text = filing.text()
         source = f"{name} 10-K (FY{year})"
         for chunk in splitter.split_text(text):
             docs.append(
@@ -71,12 +74,19 @@ def fetch_documents() -> list[Document]:
                         "ticker": ticker,
                         "company": name,
                         "year": str(year),
-                        "accession": accession,
+                        "accession": filing.accession_no,
                         "source": source,
+                        "type": "text",
                     },
                 )
             )
-        print(f"{len(text):,} chars -> {len(docs)} chunks so far")
+
+        # 2) XBRL FINANCIAL STATEMENTS -> structured facts (numeric questions).
+        fin_docs = financial_documents(filing, ticker, name, year)
+        docs.extend(fin_docs)
+
+        print(f"{len(text):,} chars + {len(fin_docs)} financial statements "
+              f"-> {len(docs)} docs so far")
         time.sleep(0.5)   # be polite to SEC's servers
     return docs
 

src/rag.py

@@ -146,13 +146,43 @@ def detect_ticker(question: str) -> str | None:
     return None
 
 
+# Terms that signal a numeric/financial question -> pull in the XBRL statements.
+_FINANCIAL_TERMS = (
+    "revenue", "sales", "income", "earnings", "profit", "margin", "ebitda",
+    "asset", "liabilit", "equity", "cash flow", "cash", "debt", "expense",
+    "eps", "per share", "how much", "dividend", "operating", "gross", "net ",
+    "balance sheet", "capital", "ratio",
+)
+
+
+def _is_financial_query(question: str) -> bool:
+    q = question.lower()
+    return any(term in q for term in _FINANCIAL_TERMS)
+
+
 def retrieve(question: str, ticker: str | None):
     store = get_vectorstore()
     search_kwargs: dict = {"k": config.TOP_K}
     if ticker:
         # Metadata filter = search ONLY that company's filings.
         search_kwargs["filter"] = {"ticker": ticker}
-    return store.as_retriever(search_kwargs=search_kwargs).invoke(question)
+    docs = store.as_retriever(search_kwargs=search_kwargs).invoke(question)
+
+    # Hybrid step: for a numeric/financial question about a known company,
+    # guarantee that company's structured XBRL statements are in context --
+    # they can otherwise be out-ranked by revenue *discussion* in the filing
+    # text (as happens for Apple).
+    if ticker and _is_financial_query(question):
+        fin = store.as_retriever(
+            search_kwargs={
+                "k": 2,
+                "filter": {"$and": [{"ticker": ticker}, {"type": "financials"}]},
+            }
+        ).invoke(question)
+        seen = {d.page_content[:80] for d in fin}
+        rest = [d for d in docs if d.page_content[:80] not in seen]
+        docs = (fin + rest)[: config.TOP_K]
+    return docs
 
 
 def format_context(docs) -> str:

vectorstore/{615fd89b-a274-475a-8189-64a3c37ef9f7 → 106fd921-0890-496a-95dd-cba23b02f735}/data_level0.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fb5e357d3efb20827f7972d9d72b0384f9301878ddd40f2d1479c5c82388bfa8
-size 35541256
+oid sha256:8c928a39e9351d4c9febd587413bbafab14c4a987e8ebee394f08804697a4cab
+size 35666956

vectorstore/{615fd89b-a274-475a-8189-64a3c37ef9f7 → 106fd921-0890-496a-95dd-cba23b02f735}/header.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0093aec8945d3efc9942dcf58966ec769316b3f1cb51497648bb773d43311c6e
+oid sha256:a5f9fd9f1648dd4e3fdfec3fa226de568f9c5af5f9bea7d5a42b7b71dddab8b7
 size 100

vectorstore/{615fd89b-a274-475a-8189-64a3c37ef9f7 → 106fd921-0890-496a-95dd-cba23b02f735}/index_metadata.pickle

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a886d4dae7bf3edd8aca30a00285319ef1bac904a694169ff526fe65ccacab67
-size 1951164
+oid sha256:3bcf185e708e3adb78620b53ea97246cb1d26fec4e928749e570bd2bab6d938c
+size 1958064

vectorstore/{615fd89b-a274-475a-8189-64a3c37ef9f7 → 106fd921-0890-496a-95dd-cba23b02f735}/length.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:19f08c1fcb54ea3c693532da8f45031a8827042407de975fdd24bdf9424661dd
-size 84824
+oid sha256:f30b58007de341e2f2ba7cb41cd5c1865f6d69e5af157a1fae4e6bccb739c94e
+size 85124

vectorstore/{615fd89b-a274-475a-8189-64a3c37ef9f7 → 106fd921-0890-496a-95dd-cba23b02f735}/link_lists.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1379a6c040ce26666a0cb509ceceacdcac173bc54f8199e661a9af5d776277f0
-size 182336
+oid sha256:b7a26b0bce7bcdf3f6e9b091cc73b09960b452087339c31b059a9a0cbd3eebf3
+size 181956

vectorstore/chroma.sqlite3

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5aaad139c84897df5458309236ca3fa8200a68df2fe85b5d550d87921d97cd92
-size 122077184
+oid sha256:4e7d02066f6535862c4db2e8ef3ce5ac23ecb2dee1a1d8e16468a7f4603b489a
+size 124530688