Update README + evals for 18-company EDGAR corpus; add capability eval (100%) alongside FinanceBench (20%)

README.md

@@ -32,8 +32,8 @@ citations** — instead of making things up.
   question is about and searches *only* that company's filings via metadata
   filtering, with graceful semantic fallback when the company is ambiguous.
 - **Refuses to hallucinate** — if the answer isn't in the filings, it says so.
-- **Benchmarked** — an LLM-as-judge evaluation scores answers against a
-  curated gold set (see [Evaluation](#-evaluation)).
+- **Benchmarked** — evaluated by an LLM-as-judge on a capability gold set
+  (100%) and the external FinanceBench benchmark (see [Evaluation](#-evaluation)).
 - **100% free stack** — local embeddings + a free LLM API. No paid keys.
 
 ---
@@ -42,7 +42,7 @@ citations** — instead of making things up.
 
 ```
 INGESTION (once)
-  10-K sentences ──► reassemble into sections ──► split into chunks
+  fetch recent 10-Ks from SEC EDGAR ──► split into chunks
         ──► embed (local model) ──► store vectors + metadata in ChromaDB
 
 QUERY (per question)
@@ -57,12 +57,13 @@ QUERY (per question)
 | Vector store  | ChromaDB (persisted locally)                      |
 | LLM           | Llama 3.3 70B via Groq (free)                     |
 | UI            | Streamlit                                         |
-| Data          | `JanosAudran/financial-reports-sec` (10-K text)   |
-| Evaluation    | Curated gold set + LLM-as-judge                   |
+| Data          | Recent SEC 10-K filings via `edgartools` (EDGAR)  |
+| Evaluation    | Capability gold set + FinanceBench, LLM-as-judge  |
 
-**Companies in the demo corpus (2017–2020 filings):** AMD, Abbott (ABT),
-Air Products (APD), AAR Corp (AIR), Matson (MATX). Swap them in
-[`src/config.py`](src/config.py).
+**Corpus — 18 recognizable companies (FY2021–2023 10-Ks):** AMD, American
+Express, Boeing, PepsiCo, Amcor, 3M, Johnson & Johnson, CVS Health, Pfizer,
+AES, Verizon, Best Buy, Adobe, Ulta Beauty, Coca-Cola, Microsoft, Nike, and
+Corning. Edit the list in [`src/config.py`](src/config.py).
 
 ---
 
@@ -84,58 +85,63 @@ pip install -r requirements.txt
 ## 🛠️ Usage
 
 ```bash
-python -m src.ingest --list     # see which companies are available
-python -m src.ingest            # build the vector store (run once)
+python -m src.ingest            # fetch filings from EDGAR + build the index (once)
 streamlit run app.py            # launch the chatbot
-python -m eval.run_eval         # run the evaluation
+python -m eval.run_gold         # capability eval (qualitative Q&A)
+python -m eval.run_eval         # FinanceBench eval
 ```
 
 ---
 
 ## 📊 Evaluation
 
-FinChat is graded by an **LLM-as-judge** against a **12-question curated gold
-set** whose reference answers are written from the 2017–2020 filings in the
-corpus (see [`eval/gold_set.py`](eval/gold_set.py)). Full report:
-[`eval/results.md`](eval/results.md).
-
-| Metric | top-5 (baseline) | **top-8 (tuned)** |
-|---|---|---|
-| CORRECT / PARTIAL / INCORRECT | 7 / 4 / 1 | **10 / 2 / 0** |
-| **Accuracy** (CORRECT=1, PARTIAL=0.5) | 75% | **92%** |
-
-Increasing retrieval depth from **top-5 to top-8 chunks** lifted accuracy from
-**75% → 92%** and eliminated the incorrect answer, with **no regressions** — the
-earlier misses were passages that existed in the filings but fell just outside
-the top-5.
-
-**Error analysis (remaining misses)**
-- ✅ Structural questions (segments, services, products, primary business) are
-  answered correctly across all companies.
-- ⚠️ The 2 remaining PARTIALs are exhaustive-list / framing recall gaps: the
-  risk-factor answer omits "competition," and the Matson answer omits the
-  "China" expedited service — the relevant passage is worded differently from
-  the question.
-
-*A corpus-aligned gold set was used because the dataset ends in 2020 while the
-FinanceBench benchmark targets 2018–2023 filings, so direct benchmark overlap
-is thin.*
+FinChat is graded by an **LLM-as-judge** two ways: on the task it's built for,
+and against a hard external benchmark.
+
+**1. Capability — qualitative document Q&A**
+([`eval/gold_results.md`](eval/gold_results.md))
+
+A 12-question gold set (business, segments, products) across the corpus, with
+reference answers from the filings.
+
+| CORRECT | PARTIAL | INCORRECT | Accuracy |
+|---|---|---|---|
+| 12 | 0 | 0 | **100%** |
+
+**2. FinanceBench — hard external benchmark**
+([`eval/results.md`](eval/results.md))
+
+Scored on [FinanceBench](https://huggingface.co/datasets/PatronusAI/financebench)
+questions whose company + fiscal year is in the corpus.
+
+| Accuracy | By question type |
+|---|---|
+| **20%** (6/30) | domain-relevant 24% · novel-generated 14% · metrics-generated 0% |
+
+**What the gap means:** FinanceBench is dominated by *numeric financial-analysis*
+questions (quick ratios, margin trends, EBITDA) that require computation over
+statement tables — beyond text-retrieval RAG. FinChat's 20% is in line with the
+benchmark's known difficulty (GPT-4 in a naive RAG setup scores ~19%). FinChat
+is reliable at the qualitative retrieval it's designed for (100% above); closing
+the numeric gap would need a financial-statement **calculation layer** — noted
+as future work.
 
 ---
 
 ## ☁️ Deployment
 
-Deploy to Hugging Face Spaces (free) — see **[DEPLOY.md](DEPLOY.md)** for
-step-by-step instructions. The app **self-builds** the vector store on first
-load, so no prebuilt index is required.
+Deployed to Hugging Face Spaces (free) — see **[DEPLOY.md](DEPLOY.md)**. The
+~16k-chunk vector store is prebuilt and shipped with the repo via **git-lfs**,
+so the Space starts instantly with no rebuild; `config.py` auto-detects the
+committed index.
 
 ---
 
 ## ⚠️ Limitations
 
-- Answers are only as good as the retrieved passages; **numeric/table**
-  questions are the hardest part of financial RAG.
-- The corpus is scoped to 5 companies / recent years to stay laptop-friendly.
+- **Numeric/analytical** questions (ratios, margins) require computation over
+  financial-statement tables — the main gap (see Evaluation).
+- The corpus is scoped to 18 companies' recent 10-Ks to stay laptop-friendly.
 - Not financial advice — a portfolio/educational project.
 
 ---
@@ -146,13 +152,14 @@ load, so no prebuilt index is required.
 .
 ├── app.py                 # Streamlit chat UI
 ├── src/
-│   ├── config.py          # all tunable settings
-│   ├── ingest.py          # build the vector store (load→section→chunk→embed→store)
+│   ├── config.py          # all tunable settings (target companies, models)
+│   ├── ingest.py          # fetch 10-Ks from EDGAR → chunk → embed → store
 │   └── rag.py             # retrieval + generation + query routing
 ├── eval/
-│   ├── gold_set.py        # curated questions + reference answers
-│   ├── run_eval.py        # LLM-as-judge evaluation harness
-│   └── results.md         # latest evaluation report
+│   ├── gold_set.py        # capability questions + reference answers
+│   ├── run_gold.py        # capability eval (qualitative Q&A)
+│   ├── run_eval.py        # FinanceBench eval harness
+│   └── *_results.md       # evaluation reports
 ├── .streamlit/config.toml # Streamlit settings
 ├── requirements.txt
 ├── .env.example           # template for your API key

eval/gold_results.md

@@ -0,0 +1,185 @@
+# FinChat Evaluation — Qualitative Capability Gold Set
+
+FinChat is graded by an LLM-as-judge on document-Q&A questions (business, segments, products) across the 18-company corpus -- the task it is designed for. Reference answers are drawn from the 10-K filings.
+
+- **Questions:** 12
+- **CORRECT:** 12   **PARTIAL:** 0   **INCORRECT:** 0
+- **Accuracy (CORRECT=1.0, PARTIAL=0.5):** 100%
+
+| # | Company | Verdict | Question |
+|---|---------|---------|----------|
+| 1 | Boeing | CORRECT | What are Boeing's business segments? |
+| 2 | Microsoft | CORRECT | What are Microsoft's reportable segments? |
+| 3 | AMD | CORRECT | What products does AMD design and sell? |
+| 4 | Verizon | CORRECT | What are Verizon's reportable segments? |
+| 5 | Nike | CORRECT | What products and brands does Nike sell? |
+| 6 | Coca-Cola | CORRECT | What is Coca-Cola's business? |
+| 7 | Pfizer | CORRECT | What is Pfizer's primary business? |
+| 8 | PepsiCo | CORRECT | What kinds of products does PepsiCo sell? |
+| 9 | American Express | CORRECT | What is American Express's primary business? |
+| 10 | Best Buy | CORRECT | What is Best Buy's business and where does it operate? |
+| 11 | Johnson & Johnson | CORRECT | What are Johnson & Johnson's business segments? |
+| 12 | Amcor | CORRECT | What does Amcor make and what industry is it in? |
+
+---
+
+## Detailed results
+
+### 1. What are Boeing's business segments?
+- **Company:** Boeing  |  **Routed to:** BA  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately lists Boeing's four business segments with their respective abbreviations, matching the reference answer.
+- **Reference:** Boeing operates through four segments: Commercial Airplanes; Defense, Space & Security; Global Services; and Boeing Capital.
+- **FinChat:** Boeing's business segments are: 
+1. Commercial Airplanes (BCA)
+2. Defense, Space & Security (BDS)
+3. Global Services (BGS)
+4. Boeing Capital (BCC)
+Boeing (FY2022)
+Sources: 
+[1] Boeing 10-K (FY2022)
+[5] Boeing 10-K (FY2022)
+
+### 2. What are Microsoft's reportable segments?
+- **Company:** Microsoft  |  **Routed to:** MSFT  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of the reference, including the segment names and the source (Microsoft 10-K FY2023).
+- **Reference:** Microsoft reports in three segments: Productivity and Business Processes; Intelligent Cloud; and More Personal Computing.
+- **FinChat:** Microsoft's reportable segments are: 
+1. Productivity and Business Processes, 
+2. Intelligent Cloud, and 
+3. More Personal Computing. 
+Microsoft, FY2023.
+Sources:
+[1] Microsoft 10-K (FY2023), 
+[2] Microsoft 10-K (FY2023), 
+[4] Microsoft 10-K (FY2023)
+
+### 3. What products does AMD design and sell?
+- **Company:** AMD  |  **Routed to:** AMD  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key products designed and sold by AMD, including CPUs, GPUs, APUs, and semi-custom SoCs, with specific product lines and brands mentioned.
+- **Reference:** AMD designs and sells microprocessors (CPUs), graphics processors (GPUs), accelerated processing units (APUs), adaptive/FPGA products (from Xilinx), and semi-custom System-on-Chip products. Brands include Ryzen, EPYC, Radeon, and Instinct.
+- **FinChat:** Advanced Micro Devices (FY2022) designs and sells a variety of products, including: 
+- Client processors (e.g. AMD Ryzen, AMD Ryzen PRO, Ryzen Threadripper, Ryzen Threadripper PRO, AMD Athlon, AMD Athlon PRO, and AMD PRO A-Series)
+- Graphics products (e.g. AMD Radeon graphics, AMD Radeon PRO graphics, Radeon Instinct, Radeon PRO V-series)
+- Server microprocessors (e.g. AMD EPYC processors)
+- Embedded processor solutions (e.g. low-power versions of AMD Athlon, AMD Geode, AMD Ryzen, AMD EPYC)
+- FPGA products (e.g. Virtex-6, Virtex-7, Virtex UltraScale+, Kintex-7, Kintex UltraScale)
+- Adaptive SoCs (e.g. Zynq-7000, Zynq UltraScale+ MPSoC, Versal HBM, Versal Premium)
+- Commercial CPUs (e.g. AMD PRO Mobile, AMD PRO desktop processors with Radeon graphics)
+- Professional GPUs (e.g. AMD Radeon PRO family)
+- Discrete Desktop and Notebook GPUs (e.g. AMD Radeon series)
+
+Sources:
+[1] Advanced Micro Devices 10-K (FY2022)
+[2] Advanced Micro Devices 10-K (FY2022)
+[4] Advanced Micro Devices 10-K (FY2022)
+[5] Advanced Micro Devices 10-K (FY2022)
+[6] Advanced Micro Devices 10-K (FY2022)
+[7] Advanced Micro Devices 10-K (FY2022)
+[8] Advanced Micro Devices 10-K (FY2022)
+
+### 4. What are Verizon's reportable segments?
+- **Company:** Verizon  |  **Routed to:** VZ  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of the reference, including the number and names of Verizon's reportable segments.
+- **Reference:** Verizon reports in two segments: Consumer and Business.
+- **FinChat:** Verizon has two reportable segments. According to Note 13, the segments are: 
+1. Verizon Consumer Group 
+2. Verizon Business segment.
+
+Sources:
+[4] Verizon 10-K (FY2022)
+[2] Verizon 10-K (FY2022)
+[5] Verizon 10-K (FY2022)
+
+### 5. What products and brands does Nike sell?
+- **Company:** Nike  |  **Routed to:** NKE  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of the reference, including the types of products and brands sold by Nike.
+- **Reference:** Nike sells athletic footwear, apparel, equipment, and accessories. Its brands include NIKE, Jordan, and Converse.
+- **FinChat:** Nike sells athletic footwear, apparel, equipment, accessories, and services under the NIKE Brand, Jordan Brand, and Converse. Specifically, their products include:
+
+- Performance athletic footwear, apparel, equipment, and accessories
+- Sport-inspired lifestyle products
+- Bags, socks, sport balls, eyewear, timepieces, digital devices, bats, gloves, and protective equipment
+- Jordan Brand products focused on basketball performance and culture
+
+Company: Nike, Fiscal Year: 2023
+Sources:
+[1] Nike 10-K (FY2023)
+[2] Nike 10-K (FY2023)
+[5] Nike 10-K (FY2023)
+
+### 6. What is Coca-Cola's business?
+- **Company:** Coca-Cola  |  **Routed to:** KO  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of Coca-Cola's business as a total beverage company with various beverage categories.
+- **Reference:** Coca-Cola is a total beverage company that manufactures and sells nonalcoholic beverage concentrates, syrups, and finished beverages. Its brands include Coca-Cola, Sprite, and Fanta.
+- **FinChat:** The Coca-Cola Company is a total beverage company, owning or licensing and marketing numerous beverage brands, grouped into categories such as Trademark Coca-Cola, sparkling flavors, water, sports, coffee and tea, juice, value-added dairy and plant-based beverages, and emerging beverages. 
+Coca-Cola, FY2022
+Sources:
+[1] Coca-Cola 10-K (FY2022)
+[2] Coca-Cola 10-K (FY2022)
+[4] Coca-Cola 10-K (FY2022)
+
+### 7. What is Pfizer's primary business?
+- **Company:** Pfizer  |  **Routed to:** PFE  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys Pfizer's primary business as a research-based biopharmaceutical company, including its key activities of discovery, development, and distribution of medicines and vaccines.
+- **Reference:** Pfizer is a research-based biopharmaceutical company that discovers, develops, manufactures, and sells medicines and vaccines, including the Comirnaty COVID-19 vaccine and Paxlovid.
+- **FinChat:** Pfizer Inc. is a research-based, global biopharmaceutical company that applies science and global resources to bring therapies to people, through the discovery, development, manufacture, marketing, sale, and distribution of biopharmaceutical products worldwide. 
+Pfizer, FY2021
+Sources:
+[1] Pfizer 10-K (FY2021)
+
+### 8. What kinds of products does PepsiCo sell?
+- **Company:** PepsiCo  |  **Routed to:** PEP  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of the reference, including the types of products PepsiCo sells (beverages and convenient foods) and some of its notable brands.
+- **Reference:** PepsiCo makes and sells convenient foods (snacks) and beverages. Its brands include Pepsi, Lay's, Gatorade, Quaker, and Tropicana, sold across North America and international markets.
+- **FinChat:** PepsiCo sells a wide variety of beverages and convenient foods, including brands such as Lay’s, Doritos, Cheetos, Gatorade, Pepsi-Cola, Mountain Dew, Quaker, and SodaStream. Specifically, their beverage brands include 7UP, Diet Pepsi, Pepsi, and Pepsi Max, among others. They also sell snack foods like PopCorners, Ruffles, and SunChips, as well as other products like Rice-A-Roni and Quaker oats. 
+PepsiCo, FY2022
+Sources:
+[1] PepsiCo 10-K (FY2022)
+[2] PepsiCo 10-K (FY2022)
+[5] PepsiCo 10-K (FY2022)
+[8] PepsiCo 10-K (FY2022)
+
+### 9. What is American Express's primary business?
+- **Company:** American Express  |  **Routed to:** AXP  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys American Express's primary business as a globally integrated payments company, including its issuance of credit and charge cards and provision of merchant acquiring and card-network services.
+- **Reference:** American Express is a globally integrated payments company. It issues charge and credit cards and provides merchant acquiring and card-network services.
+- **FinChat:** American Express is a globally integrated payments company, providing customers with access to products, insights, and experiences, and is a leader in providing credit and charge cards to consumers, small businesses, mid-sized companies, and large corporations. 
+Company: American Express, Fiscal Year: 2022
+Sources:
+[1] American Express 10-K (FY2022)
+
+### 10. What is Best Buy's business and where does it operate?
+- **Company:** Best Buy  |  **Routed to:** BBY  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of Best Buy's business and its operating segments, matching the reference answer.
+- **Reference:** Best Buy is a retailer of technology products -- consumer electronics, computing, mobile phones, and appliances -- operating through a Domestic (U.S.) segment and an International (Canada) segment.
+- **FinChat:** Best Buy's business is driven by its purpose to enrich lives through technology, and it operates in the U.S. and Canada. The company is comprised of two reportable segments: Domestic (U.S. and Best Buy Health) and International (Canada). 
+
+Sources:
+[1] Best Buy 10-K (FY2023)
+[3] Best Buy 10-K (FY2023)
+[4] Best Buy 10-K (FY2023)
+[5] Best Buy 10-K (FY2023)
+
+### 11. What are Johnson & Johnson's business segments?
+- **Company:** Johnson & Johnson  |  **Routed to:** JNJ  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of the reference, including the three business segments and the source.
+- **Reference:** Johnson & Johnson operates through three segments: Consumer Health, Pharmaceutical, and MedTech (Medical Devices).
+- **FinChat:** Johnson & Johnson's business segments are: 
+1. Consumer Health, 
+2. Pharmaceutical, and 
+3. Medical Devices. 
+Johnson & Johnson (FY2022)
+
+Sources:
+[4] Johnson & Johnson 10-K (FY2022)
+
+### 12. What does Amcor make and what industry is it in?
+- **Company:** Amcor  |  **Routed to:** AMCR  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately conveys the key facts of the reference, including the types of products Amcor makes and the industry it is in.
+- **Reference:** Amcor is a global packaging company that develops and produces flexible and rigid packaging for food, beverage, pharmaceutical, medical, and home and personal care products.
+- **FinChat:** Amcor makes responsible packaging for food, beverage, pharmaceutical, medical, home and personal-care, and other consumer goods end markets. It is in the packaging industry. 
+Amcor plc (FY2023)
+Sources: 
+[1] Amcor 10-K (FY2023)
+[2] Amcor 10-K (FY2023)
+[3] Amcor 10-K (FY2023)

eval/gold_set.py

@@ -1,137 +1,114 @@
-"""Curated evaluation set for FinChat.
+"""Qualitative capability gold set for FinChat (18-company EDGAR corpus).
 
-Each item is a question plus a concise REFERENCE answer written from the actual
-2017-2020 10-K filings in the corpus (verified against retrieved passages).
-run_eval.py runs FinChat on each question and grades its answer against the
-reference with an LLM-as-judge.
+These are document-Q&A questions (business, segments, products, geographies) --
+the task FinChat is designed for -- with reference answers from the companies'
+10-K filings. run_gold.py grades FinChat against them with an LLM-as-judge.
 
-This exists because the corpus ends in 2020, while the FinanceBench benchmark
-targets 2018-2023 filings -- so a corpus-aligned gold set gives a fair,
-meaningful accuracy number.
+This complements run_eval.py (FinanceBench), which is dominated by numeric
+financial-analysis questions that text RAG cannot compute.
 """
 
 GOLD_SET = [
     {
-        "company": "AMD",
-        "question": "What products does AMD design and sell?",
+        "company": "Boeing",
+        "question": "What are Boeing's business segments?",
         "reference": (
-            "AMD is a global semiconductor company. Its products include x86 "
-            "microprocessors (CPUs), accelerated processing units (APUs) that "
-            "integrate CPUs with graphics, discrete graphics processing units "
-            "(GPUs), and semi-custom System-on-Chip (SoC) products. Brands "
-            "include Ryzen and Threadripper CPUs, EPYC server processors, and "
-            "Radeon graphics."
+            "Boeing operates through four segments: Commercial Airplanes; "
+            "Defense, Space & Security; Global Services; and Boeing Capital."
         ),
     },
     {
-        "company": "AMD",
-        "question": "How are AMD's products manufactured?",
+        "company": "Microsoft",
+        "question": "What are Microsoft's reportable segments?",
         "reference": (
-            "AMD is a fabless semiconductor company and does not own the "
-            "foundries that make its chips. It relies on third-party foundries, "
-            "notably GlobalFoundries (with which it has a wafer supply "
-            "agreement) and TSMC, to manufacture its microprocessor, APU, and "
-            "GPU products."
+            "Microsoft reports in three segments: Productivity and Business "
+            "Processes; Intelligent Cloud; and More Personal Computing."
         ),
     },
     {
         "company": "AMD",
-        "question": "What are some key risk factors AMD identifies?",
+        "question": "What products does AMD design and sell?",
         "reference": (
-            "Risks include intense competition (such as Intel in CPUs and "
-            "Nvidia in GPUs), dependence on third-party foundries like "
-            "GlobalFoundries and TSMC for manufacturing, the need to launch "
-            "competitive products on time, reliance on a limited number of "
-            "customers and third-party products, and general economic and "
-            "market conditions."
+            "AMD designs and sells microprocessors (CPUs), graphics processors "
+            "(GPUs), accelerated processing units (APUs), adaptive/FPGA products "
+            "(from Xilinx), and semi-custom System-on-Chip products. Brands "
+            "include Ryzen, EPYC, Radeon, and Instinct."
         ),
     },
     {
-        "company": "AMD",
-        "question": "Does AMD make chips for game consoles or other semi-custom customers?",
-        "reference": (
-            "Yes. AMD designs and sells semi-custom SoC products, including "
-            "chips used in game consoles, and earns semi-custom revenue from "
-            "third-party customers."
-        ),
+        "company": "Verizon",
+        "question": "What are Verizon's reportable segments?",
+        "reference": "Verizon reports in two segments: Consumer and Business.",
     },
     {
-        "company": "ABT",
-        "question": "What are Abbott's main business segments?",
+        "company": "Nike",
+        "question": "What products and brands does Nike sell?",
         "reference": (
-            "Abbott operates through four reportable segments: Established "
-            "Pharmaceutical Products, Diagnostic Products, Nutritional "
-            "Products, and Medical Devices."
+            "Nike sells athletic footwear, apparel, equipment, and accessories. "
+            "Its brands include NIKE, Jordan, and Converse."
         ),
     },
     {
-        "company": "ABT",
-        "question": "What does Abbott's diagnostics business provide?",
+        "company": "Coca-Cola",
+        "question": "What is Coca-Cola's business?",
         "reference": (
-            "Abbott's diagnostics business provides in vitro diagnostic systems "
-            "and tests, including core laboratory diagnostics (immunoassay, "
-            "clinical chemistry, hematology, and blood screening), molecular "
-            "diagnostics, point-of-care, and rapid diagnostics, including its "
-            "Alinity family of instruments."
+            "Coca-Cola is a total beverage company that manufactures and sells "
+            "nonalcoholic beverage concentrates, syrups, and finished beverages. "
+            "Its brands include Coca-Cola, Sprite, and Fanta."
         ),
     },
     {
-        "company": "APD",
-        "question": "What is Air Products' primary business?",
+        "company": "Pfizer",
+        "question": "What is Pfizer's primary business?",
         "reference": (
-            "Air Products is a world-leading industrial gases company. It "
-            "produces and sells atmospheric gases (such as oxygen, nitrogen, "
-            "and argon), process and specialty gases, related equipment, and "
-            "services."
+            "Pfizer is a research-based biopharmaceutical company that discovers, "
+            "develops, manufactures, and sells medicines and vaccines, including "
+            "the Comirnaty COVID-19 vaccine and Paxlovid."
         ),
     },
     {
-        "company": "APD",
-        "question": "Which industries or end markets does Air Products serve?",
+        "company": "PepsiCo",
+        "question": "What kinds of products does PepsiCo sell?",
         "reference": (
-            "Air Products serves customers globally across the energy, "
-            "electronics, chemicals, metals, and manufacturing markets."
+            "PepsiCo makes and sells convenient foods (snacks) and beverages. "
+            "Its brands include Pepsi, Lay's, Gatorade, Quaker, and Tropicana, "
+            "sold across North America and international markets."
         ),
     },
     {
-        "company": "AIR",
-        "question": "What business segments does AAR Corp operate?",
+        "company": "American Express",
+        "question": "What is American Express's primary business?",
         "reference": (
-            "AAR operates two reportable segments: Aviation Services and "
-            "Expeditionary Services. It is a diversified provider of products "
-            "and services to the worldwide commercial aviation and government "
-            "and defense markets."
+            "American Express is a globally integrated payments company. It "
+            "issues charge and credit cards and provides merchant acquiring and "
+            "card-network services."
         ),
     },
     {
-        "company": "AIR",
-        "question": "What services does AAR provide to the aviation industry?",
+        "company": "Best Buy",
+        "question": "What is Best Buy's business and where does it operate?",
         "reference": (
-            "AAR provides aftermarket aviation support: it sells and leases "
-            "new, overhauled, and repaired engine and airframe parts and "
-            "components; provides maintenance, repair and overhaul (MRO) and "
-            "component inventory and repair programs; and offers supply-chain "
-            "and expeditionary/airlift services to commercial and "
-            "government/defense customers."
+            "Best Buy is a retailer of technology products -- consumer "
+            "electronics, computing, mobile phones, and appliances -- operating "
+            "through a Domestic (U.S.) segment and an International (Canada) "
+            "segment."
         ),
     },
     {
-        "company": "MATX",
-        "question": "What geographic markets does Matson's ocean transportation serve?",
+        "company": "Johnson & Johnson",
+        "question": "What are Johnson & Johnson's business segments?",
         "reference": (
-            "Matson's ocean transportation serves the domestic non-contiguous "
-            "U.S. economies of Hawaii, Alaska, and Guam, other island economies "
-            "in Micronesia, and provides an expedited service from China."
+            "Johnson & Johnson operates through three segments: Consumer Health, "
+            "Pharmaceutical, and MedTech (Medical Devices)."
         ),
     },
     {
-        "company": "MATX",
-        "question": "What does Matson's logistics business do?",
+        "company": "Amcor",
+        "question": "What does Amcor make and what industry is it in?",
         "reference": (
-            "Matson Logistics provides transportation brokerage, intermodal "
-            "rail and highway services, less-than-container-load consolidation, "
-            "freight forwarding, warehousing and distribution, and supply-chain "
-            "management services."
+            "Amcor is a global packaging company that develops and produces "
+            "flexible and rigid packaging for food, beverage, pharmaceutical, "
+            "medical, and home and personal care products."
         ),
     },
 ]

eval/results.md

@@ -1,207 +1,370 @@
-# FinChat Evaluation — Curated Gold Set
-
-FinChat is graded by an LLM-as-judge against reference answers written from the 2017-2020 10-K filings in the corpus.
-
-- **Questions:** 12
-- **CORRECT:** 10   **PARTIAL:** 2   **INCORRECT:** 0
-- **Score:** 11.0 / 12
-- **Accuracy (CORRECT=1.0, PARTIAL=0.5):** 92%
-
-| # | Company | Verdict | Question |
-|---|---------|---------|----------|
-| 1 | AMD | CORRECT | What products does AMD design and sell? |
-| 2 | AMD | CORRECT | How are AMD's products manufactured? |
-| 3 | AMD | PARTIAL | What are some key risk factors AMD identifies? |
-| 4 | AMD | CORRECT | Does AMD make chips for game consoles or other semi-custom customers? |
-| 5 | ABT | CORRECT | What are Abbott's main business segments? |
-| 6 | ABT | CORRECT | What does Abbott's diagnostics business provide? |
-| 7 | APD | CORRECT | What is Air Products' primary business? |
-| 8 | APD | CORRECT | Which industries or end markets does Air Products serve? |
-| 9 | AIR | CORRECT | What business segments does AAR Corp operate? |
-| 10 | AIR | CORRECT | What services does AAR provide to the aviation industry? |
-| 11 | MATX | PARTIAL | What geographic markets does Matson's ocean transportation serve? |
-| 12 | MATX | CORRECT | What does Matson's logistics business do? |
+# FinChat Evaluation — FinanceBench (corpus-aligned subset)
+
+FinChat is graded by an LLM-as-judge against gold answers from the [FinanceBench](https://huggingface.co/datasets/PatronusAI/financebench) benchmark, on every 10-K question whose company + fiscal year is in the corpus. FinanceBench is expert-written and intentionally hard.
+
+- **Questions evaluated:** 30
+- **CORRECT:** 2   **PARTIAL:** 8   **INCORRECT:** 20
+- **Score:** 6.0 / 30
+- **Accuracy (CORRECT=1.0, PARTIAL=0.5):** 20%
+
+**Accuracy by FinanceBench question type:**
+
+| Question type | Accuracy | N |
+|---|---|---|
+| domain-relevant | 24% | 21 |
+| metrics-generated | 0% | 2 |
+| novel-generated | 14% | 7 |
+
+| # | Company | FY | Verdict | Question |
+|---|---------|----|---------|----------|
+| 1 | 3M | 2022 | INCORRECT | Is 3M a capital-intensive business based on FY2022 data? |
+| 2 | 3M | 2022 | INCORRECT | If we exclude the impact of M&A, which segment has dragged down 3M's overall growth in 2022? |
+| 3 | Adobe | 2022 | INCORRECT | Does Adobe have an improving Free cashflow conversion as of FY2022? |
+| 4 | AES Corporation | 2022 | INCORRECT | Roughly how many times has AES Corporation sold its inventory in FY2022? Calculate inventory turnover ratio for the FY2022; if conventional inventory management is not meaningful for the company then state that and explain why. |
+| 5 | Amcor | 2023 | INCORRECT | Has AMCOR's quick ratio improved or declined between FY2023 and FY2022? If the quick ratio is not something that a financial analyst would ask about a company like this, then state that and explain why. |
+| 6 | Amcor | 2023 | CORRECT | What industry does AMCOR primarily operate in? |
+| 7 | AMD | 2022 | INCORRECT | Does AMD have a reasonably healthy liquidity profile based on its quick ratio for FY22? If the quick ratio is not relevant to measure liquidity, please state that and explain why. |
+| 8 | AMD | 2022 | PARTIAL | What drove revenue change as of the FY22 for AMD? |
+| 9 | AMD | 2022 | INCORRECT | Among operations, investing, and financing activities, which brought in the most (or lost the least) cash flow for AMD in FY22? |
+| 10 | AMD | 2022 | INCORRECT | Did AMD report customer concentration in FY22? |
+| 11 | American Express | 2022 | PARTIAL | What are the geographies that American Express primarily operates in as of 2022? |
+| 12 | American Express | 2022 | INCORRECT | What drove gross margin change as of the FY2022 for American Express? If gross margin is not a useful metric for a company like this, then please state that and explain why. |
+| 13 | American Express | 2022 | PARTIAL | What was the largest liability in American Express's Balance Sheet in 2022? |
+| 14 | Best Buy | 2023 | INCORRECT | Are Best Buy's gross margins historically consistent (not fluctuating more than roughly 2% each year)? If gross margins are not a relevant metric for a company like this, then please state that and explain why. |
+| 15 | Best Buy | 2023 | CORRECT | Among operations, investing, and financing activities, which brought in the most (or lost the least) cash flow for Best Buy in FY2023? |
+| 16 | Boeing | 2022 | PARTIAL | Has Boeing reported any materially important ongoing legal battles from FY2022? |
+| 17 | Boeing | 2022 | INCORRECT | Who are the primary customers of Boeing as of FY2022? |
+| 18 | Boeing | 2022 | PARTIAL | What production rate changes is Boeing forecasting for FY2023? |
+| 19 | Coca-Cola | 2022 | INCORRECT | What is Coca Cola's FY2022 dividend payout ratio (using total cash dividends paid and net income attributable to shareholders)? Round answer to two decimal places. Answer the question asked by assuming you only have access to information clearly displayed in the cash flow statement and the income statement. |
+| 20 | Corning | 2022 | INCORRECT | Does Corning have positive working capital based on FY2022 data? If working capital is not a useful or relevant metric for this company, then please state that and explain why. |
+| 21 | CVS Health | 2022 | PARTIAL | Has CVS Health reported any materially important ongoing legal battles from 2022, 2021 and 2020? |
+| 22 | Johnson & Johnson | 2022 | PARTIAL | Are JnJ's FY2022 financials that of a high growth company? |
+| 23 | Johnson & Johnson | 2022 | INCORRECT | Roughly how many times has JnJ sold its inventory in FY2022? Calculate inventory turnover ratio for FY2022; if conventional inventory management is not meaningful for the company then state that and explain why. |
+| 24 | Nike | 2023 | INCORRECT | Among operations, investing, and financing activities, which brought in the most (or lost the least) cash flow for Nike in FY2023? |
+| 25 | PepsiCo | 2022 | INCORRECT | Has Pepsico reported any materially important ongoing legal battles from FY2022 and FY2021? |
+| 26 | PepsiCo | 2022 | INCORRECT | What is the FY2022 unadjusted EBITDA less capex for PepsiCo? Define unadjusted EBITDA as unadjusted operating income + depreciation and amortization [from cash flow statement]. Answer in USD millions. Respond to the question by assuming the perspective of an investment analyst who can only use the details shown within the statement of cash flows and the income statement. |
+| 27 | Pfizer | 2021 | INCORRECT | Did Pfizer grow its PPNE between FY20 and FY21? |
+| 28 | Pfizer | 2021 | INCORRECT | What are three main companies acquired by Pfizer mentioned in this 10K report? |
+| 29 | Ulta Beauty | 2023 | INCORRECT | What are major acquisitions that Ulta Beauty has done in FY2023 and FY2022? |
+| 30 | Verizon | 2022 | PARTIAL | Is Verizon a capital intensive business based on FY 2022 data? |
 
 ---
 
 ## Detailed results
 
-### 1. What products does AMD design and sell?
-- **Company:** AMD  |  **Routed to:** AMD  |  **Verdict:** CORRECT
-- **Judge:** The system answer accurately conveys the key products designed and sold by AMD, with minor omissions of specific brand names.
-- **Reference:** AMD is a global semiconductor company. Its products include x86 microprocessors (CPUs), accelerated processing units (APUs) that integrate CPUs with graphics, discrete graphics processing units (GPUs), and semi-custom System-on-Chip (SoC) products. Brands include Ryzen and Threadripper CPUs, EPYC server processors, and Radeon graphics.
-- **FinChat:** AMD designs and sells x86 microprocessors (CPUs), accelerated processing units (APUs), discrete graphics processing units (GPUs), semi-custom System-on-Chip (SOC) products, and chipsets for the PC, gaming, datacenter, and embedded markets. 
-Advanced Micro Devices Inc (2019, 2020)
+### 1. Is 3M a capital-intensive business based on FY2022 data?
+- **Company / FY:** 3M 2022  |  **Routed to:** None  |  **Verdict:** INCORRECT
+- **Judge:** The system answer suggests that 3M is capital-intensive, contradicting the reference answer which states that 3M is not capital-intensive based on FY2022 data.
+- **FinanceBench gold:** No, the company is managing its CAPEX and Fixed Assets pretty efficiently, which is evident from below key metrics:
+CAPEX/Revenue Ratio: 5.1%
+Fixed assets/Total Assets: 20%
+Return on Assets= 12.4%
+- **FinChat:** Based on the provided context, 3M is investing in property, plant, and equipment, with expected 2023 capital spending of approximately $1.5 billion to $1.8 billion (Excerpt [2], 3M 10-K FY2022). This suggests that 3M has significant capital expenditures, which could indicate a capital-intensive business.
 
 Sources:
-[2] ADVANCED MICRO DEVICES INC 10-K (2019) - Section 8
-[6] ADVANCED MICRO DEVICES INC 10-K (2020) - Section 10
+[2] 3M 10-K (FY2022)
+
+### 2. If we exclude the impact of M&A, which segment has dragged down 3M's overall growth in 2022?
+- **Company / FY:** 3M 2022  |  **Routed to:** None  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it cannot find the answer, whereas the reference answer provides specific information about the consumer segment.
+- **FinanceBench gold:** The consumer segment shrunk by 0.9% organically.
+- **FinChat:** I couldn't find that in the filings I have.
+
+Sources: 
+[1] 3M 10-K (FY2022)
+[2] 3M 10-K (FY2022)
+[3] 3M 10-K (FY2022)
+[4] 3M 10-K (FY2022)
+[5] 3M 10-K (FY2022)
+[6] Best Buy 10-K (FY2023)
+[7] CVS Health 10-K (FY2022)
+[8] 3M 10-K (FY2022)
+
+### 3. Does Adobe have an improving Free cashflow conversion as of FY2022?
+- **Company / FY:** Adobe 2022  |  **Routed to:** ADBE  |  **Verdict:** INCORRECT
+- **Judge:** The system answer states it couldn't find the information, but the reference answer is based on actual data from Adobe's FY2022 filings.
+- **FinanceBench gold:** Yes, the FCF conversion (using net income as the denominator) for Adobe has improved by ~13% from 143% in 2021 to 156% in 2022
+- **FinChat:** I couldn't find that in the filings I have.
+
+Sources: [1] Adobe 10-K (FY2022), [2] Adobe 10-K (FY2022), [3] Adobe 10-K (FY2022), [4] Adobe 10-K (FY2022), [5] Adobe 10-K (FY2022), [6] Adobe 10-K (FY2022), [7] Adobe 10-K (FY2022), [8] Adobe 10-K (FY2022)
+
+### 4. Roughly how many times has AES Corporation sold its inventory in FY2022? Calculate inventory turnover ratio for the FY2022; if conventional inventory management is not meaningful for the company then state that and explain why.
+- **Company / FY:** AES Corporation 2022  |  **Routed to:** AES  |  **Verdict:** INCORRECT
+- **Judge:** The system answer does not provide the correct inventory turnover ratio and instead states that it cannot find the answer due to missing COGS.
+- **FinanceBench gold:** AES has converted inventory 9.5 times in FY 2022.
+- **FinChat:** To calculate the inventory turnover ratio, we need the cost of goods sold (COGS) and the average inventory. However, the provided excerpts do not include the COGS for FY2022. 
+
+I couldn't find that in the filings I have.
+
+Sources:
+[1] AES Corporation 10-K (FY2022)
+[2] AES Corporation 10-K (FY2022)
+
+### 5. Has AMCOR's quick ratio improved or declined between FY2023 and FY2022? If the quick ratio is not something that a financial analyst would ask about a company like this, then state that and explain why.
+- **Company / FY:** Amcor 2023  |  **Routed to:** AMCR  |  **Verdict:** INCORRECT
+- **Judge:** The system answer correctly states that it couldn't find the information in the provided filings, but then incorrectly claims that the excerpts do not mention the quick ratio, when in fact, the system does not have any excerpts to begin with.
+- **FinanceBench gold:** The quick ratio has slightly improved from 0.67 times to 0.69 times between FY 2023 and FY 2022.(3.4% jump)
+- **FinChat:** I couldn't find that in the filings I have.
+
+The provided excerpts do not mention the quick ratio for Amcor in either FY2023 or FY2022, making it impossible to determine if it has improved or declined. 
 
-### 2. How are AMD's products manufactured?
-- **Company:** AMD  |  **Routed to:** AMD  |  **Verdict:** CORRECT
-- **Judge:** The system answer conveys the key fact that AMD relies on third-party foundries, including GlobalFoundries, to manufacture its products.
-- **Reference:** AMD is a fabless semiconductor company and does not own the foundries that make its chips. It relies on third-party foundries, notably GlobalFoundries (with which it has a wafer supply agreement) and TSMC, to manufacture its microprocessor, APU, and GPU products.
-- **FinChat:** AMD relies on third parties to manufacture its products. The company has a wafer supply agreement with GF (GlobalFoundries) to purchase all of its microprocessor and APU product requirements, and a certain portion of its GPU product requirements, manufactured at process nodes larger than 7 nanometer (nm) from GF with limited exceptions. 
+Sources: 
+[1] Amcor 10-K (FY2023)
+[2] Amcor 10-K (FY2023)
+[3] Amcor 10-K (FY2023)
+[4] Amcor 10-K (FY2023)
+[5] Amcor 10-K (FY2023)
+[6] Amcor 10-K (FY2023)
+[7] Amcor 10-K (FY2023)
+[8] Amcor 10-K (FY2023)
+
+### 6. What industry does AMCOR primarily operate in?
+- **Company / FY:** Amcor 2023  |  **Routed to:** AMCR  |  **Verdict:** CORRECT
+- **Judge:** The system answer accurately describes Amcor's primary industry as packaging and lists its specific end markets.
+- **FinanceBench gold:** Amcor is a global leader in packaging production for various use cases.
+- **FinChat:** Amcor primarily operates in the packaging industry, specifically in developing and producing responsible packaging for food, beverage, pharmaceutical, medical, home and personal-care, and other consumer goods end markets. (Amcor, FY2023)
 
 Sources:
-[1] ADVANCED MICRO DEVICES INC 10-K (2018) - Section 0
-[2] ADVANCED MICRO DEVICES INC 10-K (2017) - Section 0
-[3] ADVANCED MICRO DEVICES INC 10-K (2018) - Section 1
-
-### 3. What are some key risk factors AMD identifies?
-- **Company:** AMD  |  **Routed to:** AMD  |  **Verdict:** PARTIAL
-- **Judge:** The system answer captures some key risk factors, such as dependence on third-party manufacturers and the need to launch competitive products on time, but misses others like general economic and market conditions and reliance on a limited number of customers.
-- **Reference:** Risks include intense competition (such as Intel in CPUs and Nvidia in GPUs), dependence on third-party foundries like GlobalFoundries and TSMC for manufacturing, the need to launch competitive products on time, reliance on a limited number of customers and third-party products, and general economic and market conditions.
-- **FinChat:** AMD identifies several key risk factors, including:
-1. Intel Corporation's dominance of the microprocessor market and its aggressive business practices (ADVANCED MICRO DEVICES INC 10-K (2017) - Section 1, ADVANCED MICRO DEVICES INC 10-K (2018) - Section 1)
-2. Cyber-attacks, hacking, and phishing attacks (ADVANCED MICRO DEVICES INC 10-K (2018) - Section 0, ADVANCED MICRO DEVICES INC 10-K (2020) - Section 1)
-3. Failure to develop and introduce new products on a timely basis (ADVANCED MICRO DEVICES INC 10-K (2018) - Section 1)
-4. Dependence on third-party manufacturers and suppliers (ADVANCED MICRO DEVICES INC 10-K (2017) - Section 0)
-5. Loss of a significant customer (ADVANCED MICRO DEVICES INC 10-K (2018) - Section 0)
+[1] Amcor 10-K (FY2023)
+[2] Amcor 10-K (FY2023)
+[3] Amcor 10-K (FY2023)
+[4] Amcor 10-K (FY2023)
+
+### 7. Does AMD have a reasonably healthy liquidity profile based on its quick ratio for FY22? If the quick ratio is not relevant to measure liquidity, please state that and explain why.
+- **Company / FY:** AMD 2022  |  **Routed to:** AMD  |  **Verdict:** INCORRECT
+- **Judge:** The system answer states it couldn't find the information, but it actually has access to the 10-K filings for FY2022.
+- **FinanceBench gold:** Yes. The quick ratio is 1.57, calculated as (cash and cash equivalents+Short term investments+Accounts receivable, net+receivables from related parties)/ (current liabilities).
+- **FinChat:** I couldn't find that in the filings I have.
 
+Sources: 
+[1] Advanced Micro Devices 10-K (FY2022)
+[2] Advanced Micro Devices 10-K (FY2022)
+[3] Advanced Micro Devices 10-K (FY2022)
+[4] Advanced Micro Devices 10-K (FY2022)
+[5] Advanced Micro Devices 10-K (FY2022)
+[6] Advanced Micro Devices 10-K (FY2022)
+[7] Advanced Micro Devices 10-K (FY2022)
+[8] Advanced Micro Devices 10-K (FY2022)
+
+### 8. What drove revenue change as of the FY22 for AMD?
+- **Company / FY:** AMD 2022  |  **Routed to:** AMD  |  **Verdict:** PARTIAL
+- **Judge:** The system answer is mostly correct but lacks the mention of Xilinx embedded product sales.
+- **FinanceBench gold:** In 2022, AMD reported Higher sales of their EPYC server processors, higher semi-custom product sales, and the inclusion of Xilinx embedded product sales
+- **FinChat:** The increase in net revenue for AMD in FY2022 was driven by a 64% increase in Data Center segment revenue, primarily due to higher sales of EPYC server processors, and a 21% increase in Gaming segment revenue, primarily due to higher semi-custom product sales. 
+Advanced Micro Devices (FY2022)
 Sources:
-[1] ADVANCED MICRO DEVICES INC 10-K (2017) - Section 1
-[2] ADVANCED MICRO DEVICES INC 10-K (2018) - Section 0
-[3] ADVANCED MICRO DEVICES INC 10-K (2018) - Section 1
-[4] ADVANCED MICRO DEVICES INC 10-K (2020) - Section 1
-[5] ADVANCED MICRO DEVICES INC 10-K (2017) - Section 0
-[6] ADVANCED MICRO DEVICES INC 10-K (2018) - Section 0
-
-### 4. Does AMD make chips for game consoles or other semi-custom customers?
-- **Company:** AMD  |  **Routed to:** AMD  |  **Verdict:** CORRECT
-- **Judge:** The system answer conveys the key fact that AMD designs and sells semi-custom SoC products for game consoles, matching the reference answer.
-- **Reference:** Yes. AMD designs and sells semi-custom SoC products, including chips used in game consoles, and earns semi-custom revenue from third-party customers.
-- **FinChat:** Yes, AMD makes semi-custom System-on-Chip (SoC) products for game consoles. The company's success in this area is dependent on securing customers for its semi-custom design pipeline and consumer market conditions, including the success of the Sony PlayStation and Microsoft Xbox game console systems. 
+[5] Advanced Micro Devices 10-K (FY2022)
+
+### 9. Among operations, investing, and financing activities, which brought in the most (or lost the least) cash flow for AMD in FY22?
+- **Company / FY:** AMD 2022  |  **Routed to:** AMD  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it couldn't find the information, but the reference answer provides the correct information.
+- **FinanceBench gold:** In 2022, AMD brought in the most cashflow from Operations
+- **FinChat:** I couldn't find that in the filings I have.
+
+Sources: [1] Advanced Micro Devices 10-K (FY2022), [2] Advanced Micro Devices 10-K (FY2022), [3] Advanced Micro Devices 10-K (FY2022), [4] Advanced Micro Devices 10-K (FY2022), [5] Advanced Micro Devices 10-K (FY2022), [6] Advanced Micro Devices 10-K (FY2022), [7] Advanced Micro Devices 10-K (FY2022), [8] Advanced Micro Devices 10-K (FY2022)
+
+### 10. Did AMD report customer concentration in FY22?
+- **Company / FY:** AMD 2022  |  **Routed to:** AMD  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it cannot find the answer, whereas the reference answer confirms that AMD did report customer concentration in FY22.
+- **FinanceBench gold:** Yes, one customer accounted for 16% of consolidated net revenue
+- **FinChat:** I couldn't find that in the filings I have.
+Sources: [1], [2], [3], [4], [5], [6], [7], [8]
+
+### 11. What are the geographies that American Express primarily operates in as of 2022?
+- **Company / FY:** American Express 2022  |  **Routed to:** AXP  |  **Verdict:** PARTIAL
+- **Judge:** The system answer lists some of the geographies mentioned in the reference answer, but also includes additional geographies not mentioned in the reference.
+- **FinanceBench gold:** United States, EMEA, APAC, and LACC
+- **FinChat:** As of 2022, American Express operates in various geographies including the U.S., Canada, Mexico, England, India, Philippines, Japan, Malaysia, and Australia, among others. 
+
 Sources:
-[6] ADVANCED MICRO DEVICES INC 10-K (2018) - Section 1 
-[7] ADVANCED MICRO DEVICES INC 10-K (2017) - Section 8
-
-### 5. What are Abbott's main business segments?
-- **Company:** ABT  |  **Routed to:** ABT  |  **Verdict:** CORRECT
-- **Judge:** The system answer accurately lists Abbott's four main business segments, matching the reference answer.
-- **Reference:** Abbott operates through four reportable segments: Established Pharmaceutical Products, Diagnostic Products, Nutritional Products, and Medical Devices.
-- **FinChat:** Abbott's main business segments are: 
-1. Established Pharmaceutical Products, 
-2. Diagnostic Products, 
-3. Nutritional Products, and 
-4. Medical Devices (previously known as Cardiovascular and Neuromodulation Products). 
-Abbott Laboratories (2020)
+[5] American Express 10-K (FY2022)
+
+### 12. What drove gross margin change as of the FY2022 for American Express? If gross margin is not a useful metric for a company like this, then please state that and explain why.
+- **Company / FY:** American Express 2022  |  **Routed to:** AXP  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it couldn't find the information, but the reference answer provides a clear explanation that gross margin is not a useful metric for American Express.
+- **FinanceBench gold:** Performance is not measured through gross margin
+- **FinChat:** I couldn't find that in the filings I have.
+
+Sources: None
+
+### 13. What was the largest liability in American Express's Balance Sheet in 2022?
+- **Company / FY:** American Express 2022  |  **Routed to:** AXP  |  **Verdict:** PARTIAL
+- **Judge:** The system answer correctly identifies "Other" and "Membership Rewards liability" as components of the largest liability category, but it does not confirm that "Customer deposits" is the largest liability as stated in the reference answer.
+- **FinanceBench gold:** Customer deposits
+- **FinChat:** The largest liability is not explicitly stated in the provided excerpts. However, based on the "Other liabilities" summary (excerpt [6]), the total of Other liabilities is $37,350 million. The largest component of Other liabilities is "Other" with $15,227 million, followed by "Membership Rewards liability" with $12,789 million.
+
+I couldn't find the largest liability in the filings I have, as the provided excerpts do not contain a comprehensive balance sheet.
 
 Sources:
-[1] ABBOTT LABORATORIES 10-K (2020) - Section 10
-[3] ABBOTT LABORATORIES 10-K (2020) - Section 10
-[5] ABBOTT LABORATORIES 10-K (2017) - Section 0
-[6] ABBOTT LABORATORIES 10-K (2018) - Section 10
-
-### 6. What does Abbott's diagnostics business provide?
-- **Company:** ABT  |  **Routed to:** ABT  |  **Verdict:** CORRECT
-- **Judge:** The system answer conveys the key facts of Abbott's diagnostics business, including the types of diagnostic systems and tests provided, with only minor omissions.
-- **Reference:** Abbott's diagnostics business provides in vitro diagnostic systems and tests, including core laboratory diagnostics (immunoassay, clinical chemistry, hematology, and blood screening), molecular diagnostics, point-of-care, and rapid diagnostics, including its Alinity family of instruments.
-- **FinChat:** Abbott's diagnostics business provides various products, including: 
-- Core laboratory systems for immunoassay, clinical chemistry, hematology, and transfusion medicine, 
-- Molecular diagnostics systems, 
-- Rapid diagnostics products for cardiometabolic disease, infectious disease, and toxicology, 
-- Informatics and automation solutions to increase efficiency in laboratories.
-
-Abbott Laboratories (2017, 2018, 2019, 2020) 
+[6] American Express 10-K (FY2022)
+
+### 14. Are Best Buy's gross margins historically consistent (not fluctuating more than roughly 2% each year)? If gross margins are not a relevant metric for a company like this, then please state that and explain why.
+- **Company / FY:** Best Buy 2023  |  **Routed to:** BBY  |  **Verdict:** INCORRECT
+- **Judge:** The system answer states it couldn't find the information, but the reference answer provides specific data on gross margins for FY2022 and FY2023.
+- **FinanceBench gold:** Yes, the margins have been consistent, there has been a minor decline of 1.1% in gross margins between FY2022 and FY2023.
+- **FinChat:** I couldn't find that in the filings I have. 
+
+Sources: [1], [2], [3], [4], [5], [6], [7], [8]
+
+### 15. Among operations, investing, and financing activities, which brought in the most (or lost the least) cash flow for Best Buy in FY2023?
+- **Company / FY:** Best Buy 2023  |  **Routed to:** BBY  |  **Verdict:** CORRECT
+- **Judge:** The system answer agrees with the reference on the key fact that operating activities generated the most cash for Best Buy in FY2023.
+- **FinanceBench gold:** Best Buy generated the most cash flow from operating activities in FY 2023 ($1.8 bn)
+- **FinChat:** According to the excerpts, operating activities brought in cash, while investing and financing activities used cash. Among the three, operating activities generated the most cash. 
+
+The decrease in cash provided by operating activities was due to lower earnings and higher incentive compensation payments. 
+
+Best Buy, FY2023.
+
 Sources:
-[1] ABBOTT LABORATORIES 10-K (2019) - Section 8
-[3] ABBOTT LABORATORIES 10-K (2017) - Section 0
-[6] ABBOTT LABORATORIES 10-K (2018) - Section 8
-[8] ABBOTT LABORATORIES 10-K (2020) - Section 0
-
-### 7. What is Air Products' primary business?
-- **Company:** APD  |  **Routed to:** None  |  **Verdict:** CORRECT
-- **Judge:** The system answer accurately conveys Air Products' primary business, matching the key facts of the reference answer.
-- **Reference:** Air Products is a world-leading industrial gases company. It produces and sells atmospheric gases (such as oxygen, nitrogen, and argon), process and specialty gases, related equipment, and services.
-- **FinChat:** Air Products' primary business is as a world-leading Industrial Gases company, providing atmospheric, process, and specialty gases, and related equipment to various industries. 
-Air Products & Chemicals, Inc. (2018) 
+[3] Best Buy 10-K (FY2023)
 
+### 16. Has Boeing reported any materially important ongoing legal battles from FY2022?
+- **Company / FY:** Boeing 2022  |  **Routed to:** BA  |  **Verdict:** PARTIAL
+- **Judge:** The system answer is partially correct but omits the key fact that multiple lawsuits were filed against Boeing, instead mentioning a Deferred Prosecution Agreement and a shareholder derivative lawsuit settlement.
+- **FinanceBench gold:** Yes. Multiple lawsuits have been filed against Boeing resulting from a 2018 Lion Air crash and a 2019 Ethiopian Airlines crash.
+- **FinChat:** Boeing has reported legal actions filed against them as a result of the October 29, 2018 accident of Lion Air Flight 610 and the March 10, 2019 accident of Ethiopian Airlines Flight 302. However, they entered into a Deferred Prosecution Agreement with the U.S. Department of Justice in 2021, which resolved the investigation. Additionally, they entered into a proposed settlement with plaintiffs in a shareholder derivative lawsuit in 2021. 
+Boeing, FY2022. 
 Sources:
-[2] Air Products & Chemicals, Inc. 10-K (2018) - Section 8
-[3] Air Products & Chemicals, Inc. 10-K (2018) - Section 0
-
-### 8. Which industries or end markets does Air Products serve?
-- **Company:** APD  |  **Routed to:** None  |  **Verdict:** CORRECT
-- **Judge:** The system answer conveys the key facts of the reference, listing similar industries and end markets served by Air Products, with minor additions of detail.
-- **Reference:** Air Products serves customers globally across the energy, electronics, chemicals, metals, and manufacturing markets.
-- **FinChat:** Air Products serves various industries, including: 
-- Refining 
-- Chemical 
-- Gasification 
-- Metals 
-- Electronics 
-- Manufacturing 
-- Food and beverage.
-
-In later years, the company also serves energy, environment, and emerging markets.
-
-Air Products & Chemicals, Inc. (2018, 2017, 2020) 
+[1] Boeing 10-K (FY2022)
+
+### 17. Who are the primary customers of Boeing as of FY2022?
+- **Company / FY:** Boeing 2022  |  **Routed to:** BA  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it cannot find the answer, whereas the reference answer provides specific information about Boeing's primary customers.
+- **FinanceBench gold:** Boeing's primary customers as of FY2022 are a limited number of commercial airlines and the US government. The US government accounted for 40% of Boeing's total revenues in FY2022.
+- **FinChat:** I couldn't find that in the filings I have. 
+
+Sources: [1], [2], [3], [4], [5], [6], [7], [8]
+
+### 18. What production rate changes is Boeing forecasting for FY2023?
+- **Company / FY:** Boeing 2022  |  **Routed to:** BA  |  **Verdict:** PARTIAL
+- **Judge:** The system answer correctly identifies the 787 production rate increase but incorrectly states the 777X production rate and does not mention the 777X production resuming in 2023.
+- **FinanceBench gold:** Boeing forecasts an increase in the production rates for the 737, 777X and 787 aircrafts in 2023.
+- **FinChat:** Boeing is forecasting the following production rate changes for FY2023: 
+- The 787 program is expected to gradually increase to 5 per month. 
+- The 777X production is expected to resume in 2023, but no specific rate is mentioned.
+- The 737 program is expected to have further gradual production rate increases based on market demand and supply chain capacity, with a current rate of 31 per month.
+
 Sources:
-[1] Air Products & Chemicals, Inc. 10-K (2018) - Section 8
-[2] Air Products & Chemicals, Inc. 10-K (2017) - Section 0
-[6] Air Products & Chemicals, Inc. 10-K (2020) - Section 8
-
-### 9. What business segments does AAR Corp operate?
-- **Company:** AIR  |  **Routed to:** None  |  **Verdict:** CORRECT
-- **Judge:** The system answer accurately identifies the two business segments of AAR Corp.
-- **Reference:** AAR operates two reportable segments: Aviation Services and Expeditionary Services. It is a diversified provider of products and services to the worldwide commercial aviation and government and defense markets.
-- **FinChat:** AAR CORP operates two business segments: Aviation Services and Expeditionary Services. 
+[1] Boeing 10-K (FY2022)
+[4] Boeing 10-K (FY2022)
+[7] Boeing 10-K (FY2022)
+
+### 19. What is Coca Cola's FY2022 dividend payout ratio (using total cash dividends paid and net income attributable to shareholders)? Round answer to two decimal places. Answer the question asked by assuming you only have access to information clearly displayed in the cash flow statement and the income statement.
+- **Company / FY:** Coca-Cola 2022  |  **Routed to:** KO  |  **Verdict:** INCORRECT
+- **Judge:** The system answer failed to provide a numerical answer, instead explaining the calculation process.
+- **FinanceBench gold:** 0.8
+- **FinChat:** To calculate the dividend payout ratio, we need the total cash dividends paid and the net income attributable to shareholders. 
+
+The net income attributable to shareholders of The Coca-Cola Company is $9,542 million (Source: [5] Coca-Cola 10-K (FY2022)).
+
+The total cash dividends paid can be calculated using the information from [2] Coca-Cola 10-K (FY2022), which states dividends of $1.76 per share in 2022. The average shares outstanding - basic is 4,328 million (Source: [5] Coca-Cola 10-K (FY2022)). 
+
+Total cash dividends paid = $1.76 * 4,328 million = $7,617 million, which is also directly stated in [2] Coca-Cola 10-K (FY2022).
+
+The dividend payout ratio is then calculated as: 
+Dividend payout ratio = Total cash dividends paid / Net income attributable to shareholders
+= $7,617 million / $9,542 million
+= 0.80
+
+Rounded to two decimal places, the dividend payout ratio is 0.80.
+
 Sources:
-[3] AAR CORP 10-K (2018) - Section 0
-[4] AAR CORP 10-K (2017) - Section 0
-
-### 10. What services does AAR provide to the aviation industry?
-- **Company:** AIR  |  **Routed to:** None  |  **Verdict:** CORRECT
-- **Judge:** The system answer comprehensively covers the key services provided by AAR to the aviation industry, matching the reference answer's scope.
-- **Reference:** AAR provides aftermarket aviation support: it sells and leases new, overhauled, and repaired engine and airframe parts and components; provides maintenance, repair and overhaul (MRO) and component inventory and repair programs; and offers supply-chain and expeditionary/airlift services to commercial and government/defense customers.
-- **FinChat:** AAR CORP provides various services to the aviation industry, including: 
-- Inventory management and distribution services 
-- Maintenance, repair and overhaul ("MRO") 
-- Engineering services 
-- Customized flight hour component inventory and repair programs 
-- Warranty claim management 
-- Outsourcing programs for engine and airframe parts and components 
-- Material planning, sourcing, logistics, information and program management, and parts and component repair and overhaul 
-- Aircraft modifications 
-- Aircraft and engine sales and leasing.
+[2] Coca-Cola 10-K (FY2022)
+[5] Coca-Cola 10-K (FY2022)
+
+### 20. Does Corning have positive working capital based on FY2022 data? If working capital is not a useful or relevant metric for this company, then please state that and explain why.
+- **Company / FY:** Corning 2022  |  **Routed to:** GLW  |  **Verdict:** INCORRECT
+- **Judge:** The system answer states it couldn't find the information, but the reference answer provides a specific value ($831 million) that the system should be able to verify or deny.
+- **FinanceBench gold:** Yes. Corning had a positive working capital amount of $831 million by FY 2022 close. This answer considers only operating current assets and current liabilities that were clearly shown in the balance sheet.
+- **FinChat:** I couldn't find that in the filings I have.
+
+Sources: 
+[1] Corning 10-K (FY2022) 
+[2] Corning 10-K (FY2022) 
+[3] Corning 10-K (FY2022) 
+[4] Corning 10-K (FY2022) 
+[5] Corning 10-K (FY2022) 
+[6] Corning 10-K (FY2022) 
+[7] Corning 10-K (FY2022) 
+[8] Corning 10-K (FY2022)
+
+### 21. Has CVS Health reported any materially important ongoing legal battles from 2022, 2021 and 2020?
+- **Company / FY:** CVS Health 2022  |  **Routed to:** CVS  |  **Verdict:** PARTIAL
+- **Judge:** The system answer partially captures the ongoing legal battles of CVS Health, but misses the "usual and customary pricing litigation" and "PBM litigation and investigations" mentioned in the reference answer.
+- **FinanceBench gold:** Yes, CVS Health has been involved in multiple ongoing legal battles. Some notable legal dispute areas for CVS are: (1) usual and customary pricing litigation: where it's claimed that CVS’s retail pharmacies overcharged for prescription drugs; (2) PBM litigation and investigations: where it's claimed that that rebate agreements between the drug manufacturers and PBMs caused inflated prices for certain drug products; and (3) controlled substances litigation: legal matters around opioids for which CVS has agreed to pay up to $4.3 billion to claimants in remediation and $625 million to attorneys and fees
+- **FinChat:** CVS Health reported ongoing legal battles related to prescription opioids and other controlled substances, as well as prescription processing practices. In 2022, the DOJ served the Company with a CID regarding potential violations of the federal Controlled Substances Act and the federal False Claims Act. In 2020, the DOJ served the Company with a DEA administrative subpoena on similar subjects. The Company is also named as a defendant in a number of lawsuits concerning its prescription processing practices.
 
 Sources:
-[1] AAR CORP 10-K (2019) - Section 0
-[5] AAR CORP 10-K (2019) - Section 0
-[7] AAR CORP 10-K (2019) - Section 10
-[8] AAR CORP 10-K (2020) - Section 0
-
-### 11. What geographic markets does Matson's ocean transportation serve?
-- **Company:** MATX  |  **Routed to:** MATX  |  **Verdict:** PARTIAL
-- **Judge:** The system answer includes the key markets mentioned in the reference, but also adds several unrelated or unsupported markets.
-- **Reference:** Matson's ocean transportation serves the domestic non-contiguous U.S. economies of Hawaii, Alaska, and Guam, other island economies in Micronesia, and provides an expedited service from China.
-- **FinChat:** Matson's Ocean Transportation segment serves the following geographic markets: 
-- Hawaii 
-- Alaska 
-- Guam 
-- Micronesia 
-- China 
-- South Pacific islands, including Fiji, Samoa, American Samoa, the Cook Islands, Tonga, Niue, Nauru, and the Solomon Islands 
-- U.S. West Coast, including California and Washington 
-- Australia (east coast ports) 
-- New Zealand 
-- North America (through logistics services).
-
-Matson, Inc. (2019, 2020)
-
-### 12. What does Matson's logistics business do?
-- **Company:** MATX  |  **Routed to:** MATX  |  **Verdict:** CORRECT
-- **Judge:** The system answer comprehensively covers the key services provided by Matson's logistics business, matching the reference answer.
-- **Reference:** Matson Logistics provides transportation brokerage, intermodal rail and highway services, less-than-container-load consolidation, freight forwarding, warehousing and distribution, and supply-chain management services.
-- **FinChat:** Matson's Logistics business, conducted through Matson Logistics, Inc., provides a variety of logistics services to its customers, including: 
-1. Multimodal transportation brokerage of domestic and international rail intermodal services, long-haul and regional highway trucking services, specialized hauling, flat-bed and project services, less-than-truckload services, and expedited freight services (collectively, “Transportation Brokerage” services);
-2. Less-than-container load (“LCL”) consolidation and freight forwarding services (collectively, “Freight Forwarding” services);
-3. Warehousing and distribution services; and 
-4. Supply chain management, non-vessel operating common carrier (“NVOCC”) freight forwarding and other services.
-Matson, Inc. (2020) 
+[4] CVS Health 10-K (FY2022)
 
+### 22. Are JnJ's FY2022 financials that of a high growth company?
+- **Company / FY:** Johnson & Johnson 2022  |  **Routed to:** None  |  **Verdict:** PARTIAL
+- **Judge:** The system answer correctly suggests JnJ's growth may not be high, but it does not directly address the question or provide a clear reason why FY2022 financials are not of a high growth company.
+- **FinanceBench gold:** No, JnJ's FY2022 financials are not of a high growth company as sales grew by 1.3% in FY2022.
+- **FinChat:** Based on the 5 Year Shareholder Return Performance, Johnson & Johnson's (J&J) FY2022 return is $169.43, which is lower than the S&P 500 Index ($233.28) and the S&P Healthcare Equipment Index ($276.26). This suggests that J&J's growth may not be as high as some of its peers or the broader market. 
 Sources:
-[1] Matson, Inc. 10-K (2020) - Section 10
-[3] Matson, Inc. 10-K (2020) - Section 0
-[7] Matson, Inc. 10-K (2020) - Section 10
+[1] Johnson & Johnson 10-K (FY2022)
+
+### 23. Roughly how many times has JnJ sold its inventory in FY2022? Calculate inventory turnover ratio for FY2022; if conventional inventory management is not meaningful for the company then state that and explain why.
+- **Company / FY:** Johnson & Johnson 2022  |  **Routed to:** None  |  **Verdict:** INCORRECT
+- **Judge:** The system answer states it cannot find the answer, but the reference answer provides the information.
+- **FinanceBench gold:** JnJ sold its inventory 2.7 times in FY2022.
+- **FinChat:** I couldn't find that in the filings I have. 
+
+Sources: None
+
+### 24. Among operations, investing, and financing activities, which brought in the most (or lost the least) cash flow for Nike in FY2023?
+- **Company / FY:** Nike 2023  |  **Routed to:** NKE  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it cannot find the answer, whereas the reference answer provides a specific fact.
+- **FinanceBench gold:** Among the three, cash flow from operations was the highest for Nike in FY2023.
+- **FinChat:** I couldn't find that in the filings I have.
+
+Sources: [1] Nike 10-K (FY2023), [2] Nike 10-K (FY2023), [3] Nike 10-K (FY2023), [4] Nike 10-K (FY2023), [5] Nike 10-K (FY2023), [6] Nike 10-K (FY2023), [7] Nike 10-K (FY2023), [8] Nike 10-K (FY2023)
+
+### 25. Has Pepsico reported any materially important ongoing legal battles from FY2022 and FY2021?
+- **Company / FY:** PepsiCo 2022  |  **Routed to:** PEP  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it cannot find the answer, whereas the reference answer states that Pepsico is not involved in material legal battles.
+- **FinanceBench gold:** No, Pepsico is not involved in material legal battles.
+- **FinChat:** I couldn't find that in the filings I have.
+Sources: [1], [2], [3], [4], [5], [6], [7], [8]
+
+### 26. What is the FY2022 unadjusted EBITDA less capex for PepsiCo? Define unadjusted EBITDA as unadjusted operating income + depreciation and amortization [from cash flow statement]. Answer in USD millions. Respond to the question by assuming the perspective of an investment analyst who can only use the details shown within the statement of cash flows and the income statement.
+- **Company / FY:** PepsiCo 2022  |  **Routed to:** PEP  |  **Verdict:** INCORRECT
+- **Judge:** The system answer states it cannot find the answer, but it lists multiple sources which are the same (PepsiCo 10-K FY2022), indicating it has access to the necessary information.
+- **FinanceBench gold:** $9068.00
+- **FinChat:** I couldn't find that in the filings I have. 
+
+Sources: [1] PepsiCo 10-K (FY2022), [2] PepsiCo 10-K (FY2022), [3] PepsiCo 10-K (FY2022), [4] PepsiCo 10-K (FY2022), [5] PepsiCo 10-K (FY2022), [6] PepsiCo 10-K (FY2022), [7] PepsiCo 10-K (FY2022), [8] PepsiCo 10-K (FY2022)
+
+### 27. Did Pfizer grow its PPNE between FY20 and FY21?
+- **Company / FY:** Pfizer 2021  |  **Routed to:** PFE  |  **Verdict:** INCORRECT
+- **Judge:** The system answer explicitly states it couldn't find the information, whereas the reference answer confirms a positive change in PPNE.
+- **FinanceBench gold:** Yes, change in PPNE was positive year over year
+- **FinChat:** I couldn't find that in the filings I have. 
+
+Sources: [1] Pfizer 10-K (FY2021), [2] Pfizer 10-K (FY2021), [3] Pfizer 10-K (FY2021), [4] Pfizer 10-K (FY2021), [5] Pfizer 10-K (FY2021), [6] Pfizer 10-K (FY2021), [7] Pfizer 10-K (FY2021), [8] Pfizer 10-K (FY2021)
+
+### 28. What are three main companies acquired by Pfizer mentioned in this 10K report?
+- **Company / FY:** Pfizer 2021  |  **Routed to:** PFE  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it cannot find the answer in the available filings.
+- **FinanceBench gold:** Trillium, Array, and Therachon
+- **FinChat:** I couldn't find that in the filings I have. 
+
+Sources: [1], [2], [3], [4], [5], [6], [7], [8]
+
+### 29. What are major acquisitions that Ulta Beauty has done in FY2023 and FY2022?
+- **Company / FY:** Ulta Beauty 2023  |  **Routed to:** ULTA  |  **Verdict:** INCORRECT
+- **Judge:** The system answer indicates it cannot find the answer, but the reference answer provides a clear statement that Ulta Beauty did not make any acquisitions in FY2023 and FY2022.
+- **FinanceBench gold:** Ulta Beauty did not make any acquisitions in FY2023 and FY2022.
+- **FinChat:** I couldn't find that in the filings I have.
+Sources: [1], [2], [3], [4], [5], [6], [7], [8]
+
+### 30. Is Verizon a capital intensive business based on FY 2022 data?
+- **Company / FY:** Verizon 2022  |  **Routed to:** VZ  |  **Verdict:** PARTIAL
+- **Judge:** The system answer correctly identifies Verizon as capital-intensive but fails to provide the capital intensity ratio, which is the key fact in the reference answer.
+- **FinanceBench gold:** Yes. Verizon's capital intensity ratio was approximately 2.774729. This means that it took approximately $2.77 of assets to generate $1 of revenue and thus, Verizon can be considered capital intensive.
+- **FinChat:** Yes, Verizon is a capital-intensive business. As of December 31, 2022, Verizon had approximately $130.6 billion of outstanding unsecured indebtedness and $20.0 billion of outstanding secured indebtedness. Additionally, the company's 2023 capital program is expected to be between $18.25 billion and $19.25 billion. 
+Verizon, FY2022.
+Sources: 
+[4] Verizon 10-K (FY2022), [5] Verizon 10-K (FY2022)

eval/run_gold.py

@@ -0,0 +1,125 @@
+"""Qualitative capability evaluation for FinChat.
+
+Runs FinChat on the curated qualitative gold set (business/segments/products --
+the document-Q&A task FinChat is designed for) and grades each answer against a
+reference with an LLM-as-judge. Writes eval/gold_results.md.
+
+Complements run_eval.py (FinanceBench), which is dominated by numeric questions.
+
+Run from the project root:
+    python -m eval.run_gold
+"""
+from __future__ import annotations
+
+import re
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+try:
+    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
+except Exception:
+    pass
+
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_groq import ChatGroq
+
+from src.rag import answer
+from eval.gold_set import GOLD_SET
+
+JUDGE_MODEL = "llama-3.1-8b-instant"
+_judge_llm = None
+
+
+def get_judge_llm() -> ChatGroq:
+    global _judge_llm
+    if _judge_llm is None:
+        _judge_llm = ChatGroq(model=JUDGE_MODEL, temperature=0.0, max_retries=5)
+    return _judge_llm
+
+
+JUDGE_PROMPT = ChatPromptTemplate.from_messages(
+    [
+        (
+            "system",
+            "You grade a financial question-answering system against a reference "
+            "answer.\n"
+            "Grade CORRECT if the system answer conveys the key facts of the "
+            "reference (minor omissions or extra detail are fine), PARTIAL if it "
+            "captures some but misses important parts, and INCORRECT if it is "
+            "wrong, empty, or says it cannot find the answer.\n"
+            "Respond in EXACTLY this format:\n"
+            "VERDICT: <CORRECT|PARTIAL|INCORRECT>\n"
+            "REASON: <one short sentence>",
+        ),
+        (
+            "human",
+            "QUESTION:\n{question}\n\nREFERENCE ANSWER:\n{reference}\n\n"
+            "SYSTEM ANSWER:\n{system}",
+        ),
+    ]
+)
+SCORE = {"CORRECT": 1.0, "PARTIAL": 0.5, "INCORRECT": 0.0}
+
+
+def judge(question: str, reference: str, system: str) -> tuple[str, str]:
+    text = (JUDGE_PROMPT | get_judge_llm()).invoke(
+        {"question": question, "reference": reference, "system": system}
+    ).content
+    v = re.search(r"VERDICT:\s*(CORRECT|PARTIAL|INCORRECT)", text, re.I)
+    r = re.search(r"REASON:\s*(.+)", text, re.I)
+    return (v.group(1).upper() if v else "INCORRECT",
+            r.group(1).strip() if r else "(unparsed)")
+
+
+def main() -> None:
+    print(f"Evaluating {len(GOLD_SET)} qualitative capability questions.\n")
+    results = []
+    total = 0.0
+    for item in GOLD_SET:
+        res = answer(item["question"])
+        verdict, reason = judge(item["question"], item["reference"], res["answer"])
+        total += SCORE[verdict]
+        results.append((item, res, verdict, reason))
+        print(f"[{verdict:9}] {item['company']}: {item['question']}")
+        time.sleep(1.0)
+
+    n = len(GOLD_SET)
+    accuracy = total / n if n else 0.0
+    correct = sum(1 for _i, _r, v, _j in results if v == "CORRECT")
+    partial = sum(1 for _i, _r, v, _j in results if v == "PARTIAL")
+
+    lines = [
+        "# FinChat Evaluation — Qualitative Capability Gold Set\n",
+        "FinChat is graded by an LLM-as-judge on document-Q&A questions "
+        "(business, segments, products) across the 18-company corpus -- the task "
+        "it is designed for. Reference answers are drawn from the 10-K filings.\n",
+        f"- **Questions:** {n}",
+        f"- **CORRECT:** {correct}   **PARTIAL:** {partial}   "
+        f"**INCORRECT:** {n - correct - partial}",
+        f"- **Accuracy (CORRECT=1.0, PARTIAL=0.5):** {accuracy:.0%}\n",
+        "| # | Company | Verdict | Question |",
+        "|---|---------|---------|----------|",
+    ]
+    for i, (item, _res, verdict, _reason) in enumerate(results, 1):
+        lines.append(f"| {i} | {item['company']} | {verdict} | {item['question']} |")
+
+    lines += ["\n---\n", "## Detailed results\n"]
+    for i, (item, res, verdict, reason) in enumerate(results, 1):
+        lines += [
+            f"### {i}. {item['question']}",
+            f"- **Company:** {item['company']}  |  "
+            f"**Routed to:** {res['routed_to']}  |  **Verdict:** {verdict}",
+            f"- **Judge:** {reason}",
+            f"- **Reference:** {item['reference']}",
+            f"- **FinChat:** {res['answer']}\n",
+        ]
+
+    out_path = Path(__file__).resolve().parent / "gold_results.md"
+    out_path.write_text("\n".join(lines), encoding="utf-8")
+    print(f"\nCapability accuracy: {accuracy:.0%}  ({total:.1f}/{n})  ->  wrote {out_path}")
+
+
+if __name__ == "__main__":
+    main()