wip: eval harness + public docs bundle; relocate eval docs into evals/ (#10)

Work-in-progress checkpoint of the current eval cleanup:
- relocate the 6 evals_*.md docs into evals/ (drop the evals_ prefix);
rewrite every reference (evals.md, AGENTS.md, pyproject.toml,
data/eval/README.md, evals/grade.py, upload_dbs_to_hf.py, cross-links)
- evals/ harness: grade/judge/report/run_battery changes + new
grading_prep.py, grading_merge.py, grade_workflow.js
- data/scraping_scripts: build_public_docs_bundle.py + upload/gitignore wiring
- app/ (api, config, kb_manifest) and tests tweaks
- gitignore local-only working dirs (code_review/, workshop_AIE/)


Claude-Session: https://claude.ai/code/session_01BDhiM239toao3TAmoJ1joD

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitignore

@@ -168,6 +168,8 @@ data/chroma-db**/
 evaluation_data/chroma-db**/
 # Build artifact; downloaded from HF on first run.
 data/kb/
+# Staging dir for the public docs-only bundle (build_public_docs_bundle.py).
+data/public_docs_bundle/
 data/*_md_files/
 data/full_stack_ai_engineering/
 data/beginner_python_for_ai_engineering/
@@ -234,3 +236,8 @@ data/eval/review_log_v1.md
 # so non-.jsonl files (review_log_v1.md, review_batches/*.json) can't slip in.
 eval/
 deepseek_eval_results.md
+
+# Local-only working dirs (internal review notes, workshop slide decks).
+# Never meant for the public repo; keep them from an accidental `git add -A`.
+code_review/
+workshop_AIE/

AGENTS.md

@@ -12,7 +12,7 @@ ChromaDB for vectors; Cohere for embeddings/rerank; chat model is provider-confi
 
 - [GitHub repo](https://github.com/towardsai/ai-tutor-app)
 - [Live demo — prod Space](https://huggingface.co/spaces/towardsai-tutors/ai-tutor-chatbot) · [Dev Space (private)](https://huggingface.co/spaces/towardsai-tutors/ai-tutor)
-- [Vector DB + KB bundle](https://huggingface.co/datasets/towardsai-tutors/ai-tutor-vector-db) · [Private raw JSONL data](https://huggingface.co/datasets/towardsai-tutors/ai-tutor-data)
+- [Vector DB + KB bundle](https://huggingface.co/datasets/towardsai-tutors/ai-tutor-vector-db) (private; full corpus incl. courses) · [Public docs-only bundle](https://huggingface.co/datasets/towardsai-tutors/ai-tutor-vector-db-public) (no token needed; cold-start fallback) · [Private raw JSONL data](https://huggingface.co/datasets/towardsai-tutors/ai-tutor-data)
 
 ## Where things live
 
@@ -91,11 +91,11 @@ uv run -m data.scraping_scripts.retire_source_workflow --sources KEY [--dry-run
 
 ## Evaluation
 
-**`evals.md` is the entry point** (what we evaluate, the datasets, results, remaining work); `evals/` is the harness (`run_battery` → `grade` → `report`, plus `check_triggers` and the blinded `handgrade_workbook`). Eval datasets and run results contain **real student text**: they are gitignored and ship via the private `ai-tutor-data` HF dataset (`eval/`, `eval_runs/`) — download snippet in `evals.md`. Runs cost real API money (a 4-preset bake-off ≈ $73); grading and reporting re-run offline from saved bundles for free. **Running or contributing an experiment? Follow `evals_contributing.md`** (run → upload data to HF → record an `F<N>` finding → merge; with a definition-of-done gate).
+**`evals.md` is the entry point** (what we evaluate, the datasets, results, remaining work); `evals/` is the harness (`run_battery` → `grade` → `report`, plus `check_triggers` and the blinded `handgrade_workbook`). Eval datasets and run results contain **real student text**: they are gitignored and ship via the private `ai-tutor-data` HF dataset (`eval/`, `eval_runs/`) — download snippet in `evals.md`. Runs cost real API money (a 4-preset bake-off ≈ $323; the full eval program ≈ $590 across all Gemini 3.5 Flash runs); grading and reporting re-run offline from saved bundles for free. **Running or contributing an experiment? Follow `evals/contributing.md`** (run → upload data to HF → record an `F<N>` finding → merge; with a definition-of-done gate).
 
 ## Environment variables
 
-Chat runtime: `COHERE_API_KEY` (retrieval), one chat-model provider key (`GEMINI_API_KEY`/`GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or — for open-model eval runs — `OPENROUTER_API_KEY` / `DEEPSEEK_API_KEY`), `HF_TOKEN` (first-start download). Optional: `LANGSMITH_*` (tracing), `AI_TUTOR_API_PORT`/`HOST`/`CORS_ALLOW_ORIGINS`, `AI_TUTOR_KB_DIR`, `AI_TUTOR_MEMORY_PRESET` (default memory preset; see `app/memory_presets.py`), `NEXT_PUBLIC_AI_TUTOR_API_BASE_URL`. Data workflows also need `GITHUB_TOKEN` and `GEMINI_API_KEY`/`GOOGLE_API_KEY` (context generation). See `.env.example`.
+Chat runtime: `COHERE_API_KEY` (retrieval), one chat-model provider key (`GEMINI_API_KEY`/`GOOGLE_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or — for open-model eval runs — `OPENROUTER_API_KEY` / `DEEPSEEK_API_KEY`), `HF_TOKEN` (first-start download of the full private bundle; without it, cold start falls back to the public docs-only bundle — documentation sources only, course content hidden). Optional: `LANGSMITH_*` (tracing), `AI_TUTOR_API_PORT`/`HOST`/`CORS_ALLOW_ORIGINS`, `AI_TUTOR_KB_DIR`, `AI_TUTOR_MEMORY_PRESET` (default memory preset; see `app/memory_presets.py`), `NEXT_PUBLIC_AI_TUTOR_API_BASE_URL`. Data workflows also need `GITHUB_TOKEN` and `GEMINI_API_KEY`/`GOOGLE_API_KEY` (context generation). See `.env.example`.
 
 ## Deployment
 

app/api.py

@@ -24,7 +24,7 @@ from .chat_service import (
     warm_up_retriever,
 )
 from .chat_types import ChatEvent, ChatRequest, ChatTurn
-from .kb_manifest import load_manifest_entries
+from .kb_manifest import available_source_keys, load_manifest_entries
 from .memory_presets import MEMORY_PRESETS
 from .config import (
     AVAILABLE_MODELS,
@@ -615,9 +615,16 @@ _SOURCE_VERSIONS: dict[str, dict[str, Any]] = _load_source_versions()
 
 def _source_entries() -> list[dict[str, Any]]:
     defaults = set(DEFAULT_SELECTED_SOURCES_UI)
+    # On the public docs-only bundle the course sources are absent, so hide
+    # them from the picker instead of offering sources that return nothing.
+    # `None` means the bundle is not loaded yet (advertise everything, the
+    # prod default); otherwise only advertise sources present in the bundle.
+    available = available_source_keys()
     entries: list[dict[str, Any]] = []
     for label in AVAILABLE_SOURCES_UI:
         key = SOURCE_UI_TO_KEY[label]
+        if available is not None and key not in available:
+            continue
         group = "courses" if key in COURSE_SOURCE_KEYS else "docs"
         display = SOURCE_DISPLAY_INFO.get(key, {})
         entry: dict[str, Any] = {

app/config.py

@@ -47,6 +47,14 @@ VECTOR_COLLECTION_NAME = "chroma-db-all_sources"
 DOCUMENT_DICT_PATH = f"{VECTOR_DB_DIR}/document_dict_all_sources.pkl"
 BM25_INDEX_PATH = f"{VECTOR_DB_DIR}/bm25_index_all_sources.pkl"
 CHROMA_SQLITE_PATH = f"{VECTOR_DB_DIR}/chroma.sqlite3"
+# The full bundle (all sources, incl. gated course content) lives in the
+# private repo; cold-starting from it needs an HF_TOKEN with read access. When
+# that token is missing or lacks access, we fall back to the public docs-only
+# bundle (the 9 documentation sources, no courses) built by
+# data/scraping_scripts/build_public_docs_bundle.py. Both repos share the same
+# tree layout, so only the download source changes here.
+VECTOR_DB_REPO_ID = "towardsai-tutors/ai-tutor-vector-db"
+PUBLIC_VECTOR_DB_REPO_ID = "towardsai-tutors/ai-tutor-vector-db-public"
 KB_DIR = "data/kb"
 KB_MANIFEST_PATH = f"{KB_DIR}/generated/corpus_manifest.jsonl"
 KB_INDEX_PATH = f"{KB_DIR}/wiki/index.md"
@@ -101,6 +109,85 @@ def ensure_kb_agents_md() -> None:
         os.replace(tmp_path, target)
 
 
+def _snapshot_bundle(repo_id: str, *, token: str | None) -> None:
+    """Download a vector-db/KB bundle snapshot into ``data/``.
+
+    Mutes httpx's per-file flood during the cold-start download only. The
+    GraphRAG experiment index (~150 MB) lives in the private repo for
+    reproducibility but prod does not use it; skip it on cold start. Pull it
+    explicitly to run that eval. (The public bundle has no graphrag/ tree, so
+    the pattern is simply a no-op there.)
+    """
+    from huggingface_hub import snapshot_download
+
+    httpx_logger = logging.getLogger("httpx")
+    previous_level = httpx_logger.level
+    httpx_logger.setLevel(logging.WARNING)
+    try:
+        snapshot_download(
+            repo_id=repo_id,
+            local_dir="data",
+            repo_type="dataset",
+            token=token,
+            ignore_patterns=["graphrag/**"],
+        )
+    finally:
+        httpx_logger.setLevel(previous_level)
+
+
+def _download_bundle() -> None:
+    """Download the bundle, falling back to the public docs-only one.
+
+    With a usable ``HF_TOKEN`` we pull the full private bundle (all sources,
+    incl. gated course content). When the token is absent, or present but
+    without access to the private repo, we fall back to the public docs-only
+    bundle so the app can still cold-start (documentation sources only). Any
+    non-access error (e.g. a network failure) propagates instead of silently
+    degrading to the smaller bundle.
+    """
+    try:
+        from huggingface_hub.errors import (
+            GatedRepoError,
+            RepositoryNotFoundError,
+        )
+    except ImportError:  # older huggingface_hub
+        from huggingface_hub.utils import (
+            GatedRepoError,
+            RepositoryNotFoundError,
+        )
+
+    # Resolve the token the way huggingface_hub itself does: the HF_TOKEN env
+    # var first, then a cached `huggingface-cli login`. Keying off os.getenv
+    # alone would skip a cached login that the old implicit-token download honored.
+    try:
+        from huggingface_hub import get_token
+
+        token = get_token()
+    except Exception:  # pragma: no cover - very old hub without get_token
+        token = os.getenv("HF_TOKEN") or None
+
+    if not token:
+        logger.warning(
+            "No Hugging Face token found; downloading the public docs-only "
+            "bundle from %s (documentation sources only, no course content).",
+            PUBLIC_VECTOR_DB_REPO_ID,
+        )
+        _snapshot_bundle(PUBLIC_VECTOR_DB_REPO_ID, token=None)
+        return
+
+    try:
+        _snapshot_bundle(VECTOR_DB_REPO_ID, token=token)
+    except (GatedRepoError, RepositoryNotFoundError) as exc:
+        logger.warning(
+            "HF_TOKEN cannot access the private bundle %s (%s); falling back to "
+            "the public docs-only bundle %s.",
+            VECTOR_DB_REPO_ID,
+            type(exc).__name__,
+            PUBLIC_VECTOR_DB_REPO_ID,
+        )
+        _snapshot_bundle(PUBLIC_VECTOR_DB_REPO_ID, token=None)
+
+
 def ensure_local_vector_db() -> None:
     """Make sure the vector-db/KB bundle and AGENTS.md exist locally.
 
@@ -119,24 +206,7 @@ def ensure_local_vector_db() -> None:
                 "Vector database missing or incomplete locally, downloading "
                 "from Hugging Face"
             )
-            from huggingface_hub import snapshot_download
-
-            # Mute httpx's per-file flood during the cold-start download only.
-            httpx_logger = logging.getLogger("httpx")
-            previous_level = httpx_logger.level
-            httpx_logger.setLevel(logging.WARNING)
-            try:
-                snapshot_download(
-                    repo_id="towardsai-tutors/ai-tutor-vector-db",
-                    local_dir="data",
-                    repo_type="dataset",
-                    # The GraphRAG experiment index (~150 MB) lives in the same
-                    # repo for reproducibility but prod does not use it; skip it
-                    # on cold start. Pull it explicitly to run that eval.
-                    ignore_patterns=["graphrag/**"],
-                )
-            finally:
-                httpx_logger.setLevel(previous_level)
+            _download_bundle()
         ensure_kb_agents_md()
         _BUNDLE_READY = _bundle_complete() and os.path.exists(KB_AGENTS_PATH)
 

app/kb_manifest.py

@@ -117,6 +117,21 @@ def load_manifest_entries(kb_dir: str = KB_DIR) -> tuple[KbManifestEntry, ...]:
     return result
 
 
+def available_source_keys(kb_dir: str = KB_DIR) -> frozenset[str] | None:
+    """Source keys actually present in the downloaded KB bundle.
+
+    Returns ``None`` when the manifest is not available yet (the bundle may
+    still be downloading on first start), signalling callers to not filter.
+    Otherwise it is the distinct set of sources shipped in the bundle. On the
+    public docs-only bundle this excludes the course sources, so the UI source
+    picker can hide sources that have no data to query.
+    """
+    entries = load_manifest_entries(kb_dir)
+    if not entries:
+        return None
+    return frozenset(entry.source for entry in entries if entry.source)
+
+
 def manifest_indexes(
     kb_dir: str = KB_DIR,
 ) -> tuple[

data/eval/README.md

@@ -41,7 +41,7 @@ Built from `data/academy_discussion_eval.jsonl` — real student posts from the
 - **requires_notebook**: fully answering needs the Colab notebook contents (cells/outputs/files), which our KB does not contain — verified against lesson markdown with file/line evidence (`notebook_evidence`), not guessed. All 60 included cases are `false`; the one true case in review was excluded. If retrieval misses on a flagged-adjacent case, check `notebook_evidence` before blaming the retriever.
 - **requires_media**: the post referenced a screenshot we don't have. The 2 included cases remain gradeable without it.
 - **planted facts**: personal facts (environment, goal, level, weak topic, preference/constraint) a session's student states in turn 0, which later probes test. The session-level analog of a profile.
-- **Probe types** (`probe_type`): `fact_recall` means the answer must use a fact the student gave many turns ago. `preference_compliance` means the tutor must honor a stated preference, such as OS, tooling, language, or style. `anaphora` means the user asks about "that thing from earlier," so the tutor must resolve the reference. `anaphora_consistency` means the answer must stay consistent with the tutor's own earlier recommendation. `fact_update` means the student changed a fact mid-session; using the old value fails. `behavior_routing` means a support/platform issue appears inside a long chat, and the tutor must still redirect appropriately. Battery v2 (planned, not built) will add `contradiction` (a fact changed early and probed only after it would be evicted), `longhorizon_recall`, and `entity_isolation` (two entities in one session, no fact bleed). `cross_session_recall` was an earlier idea for `sleeptime_consolidation` and is deferred with the multi-thread battery; the v1 types here are frozen. See `evals_part_c_plan.md` → "Battery v2" for the build plan.
+- **Probe types** (`probe_type`): `fact_recall` means the answer must use a fact the student gave many turns ago. `preference_compliance` means the tutor must honor a stated preference, such as OS, tooling, language, or style. `anaphora` means the user asks about "that thing from earlier," so the tutor must resolve the reference. `anaphora_consistency` means the answer must stay consistent with the tutor's own earlier recommendation. `fact_update` means the student changed a fact mid-session; using the old value fails. `behavior_routing` means a support/platform issue appears inside a long chat, and the tutor must still redirect appropriately. Battery v2 (planned, not built) will add `contradiction` (a fact changed early and probed only after it would be evicted), `longhorizon_recall`, and `entity_isolation` (two entities in one session, no fact bleed). `cross_session_recall` was an earlier idea for `sleeptime_consolidation` and is deferred with the multi-thread battery; the v1 types here are frozen. See `evals/part_c_plan.md` → "Battery v2" for the build plan.
 - **expected_facts / check_note**: per probe, the specific facts a correct answer must use, and a one-line pass/fail rule for the grader. Grade probes binary → **memory probe accuracy**, reported overall and per probe type per memory preset.
 - **profile_seed / facts**: the persona's stored profile. Seed it with `app.chat_service.set_student_profile(student_id, profile_seed)` before running that persona's questions; `facts` is the same content structured, for slicing results by fact type.
 - **Self-grading checks** (`checks` / `anti_patterns`): persona questions grade themselves because we authored the ground truth. `{"type": "regex_any", "patterns": [...]}` = case-insensitive regex over the answer; any match passes that check. `{"type": "llm", "instruction": ...}` = a judge call with that instruction (use a strong Anthropic model; binary verdict). `anti_patterns` = case-insensitive regexes whose match FAILS the question (e.g. bash `export` advice for a Windows persona). A question passes when every check passes and no anti-pattern matches.

data/scraping_scripts/README.md

@@ -270,6 +270,37 @@ uv run -m data.scraping_scripts.retire_source_workflow --sources 8-hour_primer -
 Use `--keep-source-registry` only if you want to remove existing chunks while
 leaving the source configured as active for a future rebuild.
 
+## Publishing the public docs-only bundle
+
+The production bundle in `towardsai-tutors/ai-tutor-vector-db` is **private** and
+holds every source, including gated course content, so cold-starting the app
+needs an `HF_TOKEN` with read access. To let anyone run the chatbot without that
+token, we publish a **public, docs-only** bundle to
+`towardsai-tutors/ai-tutor-vector-db-public` (the 9 documentation/reference
+sources, no courses). The runtime (`app.config.ensure_local_vector_db`) downloads
+the full private bundle when a usable `HF_TOKEN` is present and falls back to the
+public one otherwise; the picker hides course sources when running on it.
+
+This is a **standalone, manual** step — it is **not** wired into the docs/course
+workflows. Refresh the public bundle whenever the docs sources change materially:
+
+```bash
+# Refresh the prod bundle first (so data/chroma-db-all_sources, data/kb,
+# data/all_sources_data.jsonl are current), then:
+uv run -m data.scraping_scripts.build_public_docs_bundle              # build + upload
+uv run -m data.scraping_scripts.build_public_docs_bundle --skip-upload  # build only, inspect first
+```
+
+It **derives** the docs-only bundle from the already-built prod artifacts (no
+re-embedding, $0 Cohere): it copies the Chroma collection and deletes the course
+chunks, rebuilds the BM25 / document-dict pkls from a docs-only view of
+`all_sources_data.jsonl`, and copies `data/kb` while pruning `raw/courses`,
+`wiki/courses`, and the course rows from the generated indexes. The staged tree
+lives in `data/public_docs_bundle/` (gitignored). Uploading needs an `HF_TOKEN`
+with write access to the `towardsai-tutors` org; the script creates the public
+repo on first run. Pass `--include-contextual` to also ship a docs-only
+`all_sources_contextual_nodes.pkl` (off by default to keep the download small).
+
 ## Tips for New Team Members
 
 1. To update the AI Tutor with new content:

data/scraping_scripts/build_public_docs_bundle.py

@@ -0,0 +1,380 @@
+"""Build (and optionally upload) the PUBLIC, docs-only vector-db/KB bundle.
+
+The production bundle lives in the **private** dataset
+``towardsai-tutors/ai-tutor-vector-db`` and contains every source, including
+the course content (real student-facing material we keep gated). Anyone
+without an ``HF_TOKEN`` that can read that repo cannot cold-start the app.
+
+This script derives a **public** docs-only bundle from the already-built
+production artifacts and pushes it to ``towardsai-tutors/ai-tutor-vector-db-public``
+(public). The runtime (``app.config.ensure_local_vector_db``) falls back to it
+when no usable ``HF_TOKEN`` is present. It contains only the 9 documentation /
+reference sources (``DOC_SOURCE_KEYS``); the 5 course sources are stripped.
+
+Why "derive", not "rebuild": the docs-only bundle is exactly what prod would
+produce if the course sources had never existed, so we build each artifact the
+same way prod does, minus the course rows:
+
+* **Dense (Chroma)** - copy the prod collection and ``delete`` the course
+  chunks. This reuses the existing Cohere embeddings, so the build costs **$0**
+  and is identical to prod for every docs chunk.
+* **BM25 + document dict** - rebuilt from a docs-only view of
+  ``all_sources_data.jsonl`` with the same helpers prod uses
+  (``build_document_dict`` / ``BM25Index.build``). Pure, no network.
+* **KB** - copy ``data/kb`` and prune: drop ``raw/courses`` and ``wiki/courses``
+  and filter the course rows out of the generated indexes. The rest of the
+  docs wiki is kept verbatim.
+
+Usage::
+
+    # build into data/public_docs_bundle/ and upload to the public repo
+    uv run -m data.scraping_scripts.build_public_docs_bundle
+
+    # build only (inspect locally first), skip the upload
+    uv run -m data.scraping_scripts.build_public_docs_bundle --skip-upload
+
+Prerequisites: a complete local prod bundle (``data/chroma-db-all_sources/``,
+``data/all_sources_data.jsonl``, ``data/kb/``). Build/refresh it with the docs
+workflow first. Uploading additionally needs an ``HF_TOKEN`` with write access
+to the ``towardsai-tutors`` org.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import gc
+import json
+import logging
+import shutil
+import sqlite3
+from pathlib import Path
+
+import chromadb
+
+try:
+    from data.scraping_scripts.source_registry import (
+        ALL_SOURCES_JSONL,
+        CONTEXTUAL_NODES_PKL,
+        COURSE_SOURCE_KEYS,
+        DOC_SOURCE_KEYS,
+    )
+    from app.chroma_rag import (
+        BM25Index,
+        build_chunk_records,
+        build_document_dict,
+        get_chunk_record_source,
+        load_jsonl_documents,
+        save_bm25_index,
+        save_document_dict,
+    )
+except ModuleNotFoundError:
+    import sys
+
+    project_root = Path(__file__).resolve().parents[2]
+    if str(project_root) not in sys.path:
+        sys.path.insert(0, str(project_root))
+    from data.scraping_scripts.source_registry import (
+        ALL_SOURCES_JSONL,
+        CONTEXTUAL_NODES_PKL,
+        COURSE_SOURCE_KEYS,
+        DOC_SOURCE_KEYS,
+    )
+    from app.chroma_rag import (
+        BM25Index,
+        build_chunk_records,
+        build_document_dict,
+        get_chunk_record_source,
+        load_jsonl_documents,
+        save_bm25_index,
+        save_document_dict,
+    )
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+# Layout mirrors the prod bundle so the runtime needs no path changes: the
+# public repo has the identical tree (chroma-db-all_sources/, kb/), just with
+# the course rows removed. Keep these names in sync with app/config.py.
+VECTOR_DB_DIR = "chroma-db-all_sources"
+VECTOR_COLLECTION_NAME = "chroma-db-all_sources"
+DOCUMENT_DICT_FILE = "document_dict_all_sources.pkl"
+BM25_INDEX_FILE = "bm25_index_all_sources.pkl"
+KB_DIR_NAME = "kb"
+
+PUBLIC_REPO_ID = "towardsai-tutors/ai-tutor-vector-db-public"
+DEFAULT_SOURCE_DIR = Path("data")
+DEFAULT_STAGE_DIR = Path("data/public_docs_bundle")
+
+_COURSE_KEYS = frozenset(COURSE_SOURCE_KEYS)
+
+
+def _is_course_row(row: dict) -> bool:
+    return str(row.get("source") or "") in _COURSE_KEYS
+
+
+def stage_chroma(source_dir: Path, stage_dir: Path, *, vacuum: bool = True) -> dict:
+    """Copy the prod Chroma collection and delete the course chunks.
+
+    Reuses the existing docs embeddings (no Cohere call). Returns counts for
+    the build summary.
+    """
+    src = source_dir / VECTOR_DB_DIR
+    dst = stage_dir / VECTOR_DB_DIR
+    if not src.is_dir():
+        raise SystemExit(f"Missing prod Chroma collection: {src}")
+
+    logger.info("Copying Chroma collection %s -> %s", src, dst)
+    shutil.copytree(src, dst)
+
+    client = chromadb.PersistentClient(path=str(dst))
+    collection = client.get_collection(name=VECTOR_COLLECTION_NAME)
+    before = collection.count()
+    course_keys = sorted(_COURSE_KEYS)
+    if course_keys:
+        collection.delete(where={"source": {"$in": course_keys}})
+    after = collection.count()
+    logger.info(
+        "Chroma: kept %s docs chunks, removed %s course chunks",
+        after,
+        before - after,
+    )
+
+    # Release the client so the sqlite file is unlocked before VACUUM.
+    del collection
+    del client
+    try:
+        chromadb.api.client.SharedSystemClient.clear_system_cache()
+    except Exception:  # API location varies across chromadb versions
+        pass
+    gc.collect()
+
+    if vacuum:
+        sqlite_path = dst / "chroma.sqlite3"
+        try:
+            con = sqlite3.connect(sqlite_path)
+            con.execute("VACUUM")
+            con.close()
+            logger.info("Vacuumed %s to reclaim space from deleted rows", sqlite_path)
+        except sqlite3.Error as exc:  # best-effort; a larger file is harmless
+            logger.warning("VACUUM skipped (%s); bundle will be larger", exc)
+
+    return {"dense_kept": after, "dense_removed": before - after}
+
+
+def rebuild_retrieval_pkls(source_dir: Path, stage_dir: Path) -> dict:
+    """Rebuild the BM25 index and document dict from a docs-only JSONL.
+
+    Mirrors ``create_vector_stores.write_retrieval_artifacts`` so the public
+    pkls are byte-for-byte what prod would write without the course rows.
+    """
+    jsonl_path = source_dir / Path(ALL_SOURCES_JSONL).name
+    if not jsonl_path.exists():
+        raise SystemExit(f"Missing aggregate corpus: {jsonl_path}")
+
+    all_rows = load_jsonl_documents(str(jsonl_path))
+    docs_rows = [row for row in all_rows if not _is_course_row(row)]
+    logger.info(
+        "Corpus: %s docs documents (dropped %s course documents)",
+        len(docs_rows),
+        len(all_rows) - len(docs_rows),
+    )
+
+    dst = stage_dir / VECTOR_DB_DIR
+    save_document_dict(build_document_dict(docs_rows), str(dst / DOCUMENT_DICT_FILE))
+    bm25_records = build_chunk_records(docs_rows)
+    save_bm25_index(BM25Index.build(bm25_records), str(dst / BM25_INDEX_FILE))
+    logger.info(
+        "Rebuilt document dict (%s docs) and BM25 index (%s chunks)",
+        len(docs_rows),
+        len(bm25_records),
+    )
+    return {"documents": len(docs_rows), "bm25_chunks": len(bm25_records)}
+
+
+def stage_contextual_nodes(source_dir: Path, stage_dir: Path) -> int:
+    """Write a docs-only copy of the contextual-nodes pickle (opt-in).
+
+    Not needed at runtime; included only when ``--include-contextual`` is
+    passed so others can rebuild the docs embeddings from scratch.
+    """
+    import pickle
+
+    src = source_dir / Path(CONTEXTUAL_NODES_PKL).name
+    if not src.exists():
+        logger.warning("No contextual nodes pickle at %s; skipping", src)
+        return 0
+    with src.open("rb") as handle:
+        records = pickle.load(handle)
+    kept = []
+    for record in records:
+        try:
+            record_source = get_chunk_record_source(record)
+        except Exception:
+            kept.append(record)  # keep records we cannot classify
+            continue
+        if record_source not in _COURSE_KEYS:
+            kept.append(record)
+    dst = stage_dir / Path(CONTEXTUAL_NODES_PKL).name
+    with dst.open("wb") as handle:
+        pickle.dump(kept, handle)
+    logger.info("Contextual nodes: kept %s docs chunks", len(kept))
+    return len(kept)
+
+
+def _filter_jsonl(path: Path) -> int:
+    """Drop course rows from a generated *.jsonl index in place."""
+    if not path.exists():
+        return 0
+    kept = []
+    with path.open("r", encoding="utf-8") as handle:
+        for line in handle:
+            if not line.strip():
+                continue
+            row = json.loads(line)
+            if str(row.get("source") or "") not in _COURSE_KEYS:
+                kept.append(line if line.endswith("\n") else line + "\n")
+    with path.open("w", encoding="utf-8") as handle:
+        handle.writelines(kept)
+    return len(kept)
+
+
+def _filter_tsv(path: Path) -> int:
+    """Drop course rows from symbols.tsv in place (keeps the header)."""
+    if not path.exists():
+        return 0
+    with path.open("r", encoding="utf-8", newline="") as handle:
+        reader = csv.DictReader(handle, delimiter="\t")
+        fieldnames = reader.fieldnames or []
+        kept = [
+            row for row in reader if str(row.get("source") or "") not in _COURSE_KEYS
+        ]
+    with path.open("w", encoding="utf-8", newline="") as handle:
+        writer = csv.DictWriter(handle, fieldnames=fieldnames, delimiter="\t")
+        writer.writeheader()
+        writer.writerows(kept)
+    return len(kept)
+
+
+def stage_kb(source_dir: Path, stage_dir: Path) -> dict:
+    """Copy data/kb and prune course content (raw, wiki, generated indexes)."""
+    src = source_dir / KB_DIR_NAME
+    dst = stage_dir / KB_DIR_NAME
+    if not src.is_dir():
+        raise SystemExit(f"Missing KB directory: {src}")
+
+    logger.info("Copying KB %s -> %s", src, dst)
+    shutil.copytree(src, dst)
+
+    for relative in ("raw/courses", "wiki/courses"):
+        target = dst / relative
+        if target.exists():
+            shutil.rmtree(target)
+            logger.info("Removed %s", target)
+
+    generated = dst / "generated"
+    manifest_rows = _filter_jsonl(generated / "corpus_manifest.jsonl")
+    heading_rows = _filter_jsonl(generated / "headings.jsonl")
+    symbol_rows = _filter_tsv(generated / "symbols.tsv")
+    logger.info(
+        "KB generated indexes: %s manifest rows, %s heading rows, %s symbol rows",
+        manifest_rows,
+        heading_rows,
+        symbol_rows,
+    )
+    return {"manifest_rows": manifest_rows}
+
+
+def build_bundle(
+    *,
+    source_dir: Path = DEFAULT_SOURCE_DIR,
+    stage_dir: Path = DEFAULT_STAGE_DIR,
+    include_contextual: bool = False,
+    vacuum: bool = True,
+) -> dict:
+    if stage_dir.exists():
+        logger.info("Clearing existing staging dir %s", stage_dir)
+        shutil.rmtree(stage_dir)
+    stage_dir.mkdir(parents=True, exist_ok=True)
+
+    summary: dict = {"sources": sorted(DOC_SOURCE_KEYS)}
+    summary.update(stage_chroma(source_dir, stage_dir, vacuum=vacuum))
+    summary.update(rebuild_retrieval_pkls(source_dir, stage_dir))
+    summary.update(stage_kb(source_dir, stage_dir))
+    if include_contextual:
+        summary["contextual_chunks"] = stage_contextual_nodes(source_dir, stage_dir)
+    return summary
+
+
+def public_allow_patterns(*, include_contextual: bool) -> list[str]:
+    patterns = [f"{VECTOR_DB_DIR}/**", f"{KB_DIR_NAME}/**"]
+    if include_contextual:
+        patterns.append(Path(CONTEXTUAL_NODES_PKL).name)
+    return patterns
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Build and upload the public docs-only vector-db/KB bundle."
+    )
+    parser.add_argument("--source-dir", default=str(DEFAULT_SOURCE_DIR))
+    parser.add_argument("--stage-dir", default=str(DEFAULT_STAGE_DIR))
+    parser.add_argument("--repo", default=PUBLIC_REPO_ID)
+    parser.add_argument(
+        "--skip-upload",
+        action="store_true",
+        help="Build the bundle locally but do not push it to Hugging Face.",
+    )
+    parser.add_argument(
+        "--include-contextual",
+        action="store_true",
+        help="Also ship a docs-only all_sources_contextual_nodes.pkl so others "
+        "can rebuild the docs embeddings (larger download; off by default).",
+    )
+    parser.add_argument(
+        "--no-vacuum",
+        action="store_true",
+        help="Skip the sqlite VACUUM that reclaims space from deleted rows.",
+    )
+    args = parser.parse_args()
+
+    stage_dir = Path(args.stage_dir)
+    summary = build_bundle(
+        source_dir=Path(args.source_dir),
+        stage_dir=stage_dir,
+        include_contextual=args.include_contextual,
+        vacuum=not args.no_vacuum,
+    )
+
+    print("\nBuilt public docs-only bundle:")
+    print(f"  sources:        {', '.join(summary['sources'])}")
+    print(f"  documents:      {summary['documents']}")
+    print(
+        f"  dense chunks:   {summary['dense_kept']} (removed {summary['dense_removed']})"
+    )
+    print(f"  bm25 chunks:    {summary['bm25_chunks']}")
+    print(f"  KB manifest:    {summary['manifest_rows']} rows")
+    if "contextual_chunks" in summary:
+        print(f"  contextual:     {summary['contextual_chunks']} chunks")
+    print(f"  staged at:      {stage_dir}")
+
+    if args.skip_upload:
+        print("\n--skip-upload set; not pushing to Hugging Face.")
+        return
+
+    from data.scraping_scripts.upload_dbs_to_hf import upload_bundle
+
+    print(f"\nUploading {stage_dir} -> {args.repo} (public) ...")
+    upload_bundle(
+        args.repo,
+        folder_path=str(stage_dir),
+        allow_patterns=public_allow_patterns(
+            include_contextual=args.include_contextual
+        ),
+        create_public=True,
+    )
+    print("Done.")
+
+
+if __name__ == "__main__":
+    main()

data/scraping_scripts/upload_dbs_to_hf.py

@@ -1,6 +1,7 @@
 """Upload the Chroma vector database to a Hugging Face dataset repository."""
 
 import argparse
+import os
 from pathlib import Path
 
 from dotenv import load_dotenv
@@ -82,21 +83,45 @@ def _prune_stale_remote_files(
     )
 
 
-def upload_vector_db(repo_id: str = DEFAULT_REPO_ID) -> None:
-    try:
-        api = validate_hf_access(repo_id=repo_id)
-    except HuggingFaceAuthError as exc:
-        print(exc)
-        raise SystemExit(1) from exc
+def upload_bundle(
+    repo_id: str = DEFAULT_REPO_ID,
+    *,
+    folder_path: str = FOLDER_PATH,
+    allow_patterns: list[str] | None = None,
+    ignore_patterns: list[str] | None = None,
+    create_public: bool = False,
+) -> None:
+    """Prune stale remote files, then upload ``folder_path`` to ``repo_id``.
+
+    The default arguments reproduce the production upload (the private
+    all-sources bundle). ``build_public_docs_bundle`` reuses this with a
+    different folder/repo/patterns and ``create_public=True`` to publish the
+    docs-only bundle to a public dataset repo it may need to create first.
+    """
+    allow = allow_patterns if allow_patterns is not None else ALLOW_PATTERNS
+    ignore = ignore_patterns if ignore_patterns is not None else IGNORE_PATTERNS
+
+    if create_public:
+        # The public repo may not exist yet, so we cannot validate access to it
+        # up front (validate_hf_access requires the repo to exist). Create it as
+        # a public dataset (idempotent), which also proves the token can write.
+        api = HfApi(token=os.getenv("HF_TOKEN"))
+        api.create_repo(repo_id, repo_type="dataset", private=False, exist_ok=True)
+    else:
+        try:
+            api = validate_hf_access(repo_id=repo_id)
+        except HuggingFaceAuthError as exc:
+            print(exc)
+            raise SystemExit(1) from exc
 
     # Step 1: prune. Restore the auto-cleanup we lost when moving off
     # `upload_folder(..., delete_patterns=["*"])`. See helper docstring.
     _prune_stale_remote_files(
         api,
         repo_id,
-        folder_path=FOLDER_PATH,
-        allow_patterns=ALLOW_PATTERNS,
-        ignore_patterns=IGNORE_PATTERNS,
+        folder_path=folder_path,
+        allow_patterns=allow,
+        ignore_patterns=ignore,
     )
 
     # Step 2: upload. `upload_large_folder` is the recommended path once the
@@ -105,14 +130,18 @@ def upload_vector_db(repo_id: str = DEFAULT_REPO_ID) -> None:
     # `upload_folder` tends to hit at this size. It is resumable: re-running
     # after a failure skips already-uploaded blobs.
     api.upload_large_folder(
-        folder_path=FOLDER_PATH,
+        folder_path=folder_path,
         repo_id=repo_id,
         repo_type="dataset",
-        allow_patterns=ALLOW_PATTERNS,
-        ignore_patterns=IGNORE_PATTERNS,
+        allow_patterns=allow,
+        ignore_patterns=ignore,
     )
 
 
+def upload_vector_db(repo_id: str = DEFAULT_REPO_ID) -> None:
+    upload_bundle(repo_id)
+
+
 GRAPHRAG_LOCAL_DIR = "data/graphrag/output"
 GRAPHRAG_REPO_PATH = "graphrag/output"
 
@@ -126,7 +155,7 @@ def upload_graphrag_index(repo_id: str = DEFAULT_REPO_ID) -> None:
     it must never delete the production bundle. The runtime cold-start download
     (`config.ensure_local_vector_db`) ignores `graphrag/**`, so prod Spaces do
     not pull this ~150 MB experiment artifact; pull it explicitly to run the eval
-    (see evals_graphrag.md).
+    (see evals/graphrag.md).
     """
     try:
         api = validate_hf_access(repo_id=repo_id)

evals.md

@@ -2,9 +2,9 @@
 
 This document explains how we test the AI tutor with repeatable conversations and measured results. The main question is: when the tutor has a long chat history, which memory strategy gives the best answers without wasting too many tokens, dollars, or seconds?
 
-The eval setup compares answer quality, memory retention, retrieval accuracy, tokens, cost, and latency for the June 2026 workshop. The same harness also becomes the ongoing quality program afterwards. Research sources and design rationale live in `evals_background.md`.
+The eval setup compares answer quality, memory retention, retrieval accuracy, tokens, cost, and latency for the June 2026 workshop. The same harness also becomes the ongoing quality program afterwards. Research sources and design rationale live in `evals/background.md`.
 
-**Running a new experiment?** See `evals_contributing.md` for the step-by-step (run → upload data to HF → record an `F<N>` finding here → merge) and the definition-of-done checklist, so the next person inherits both your result and the data to reproduce it.
+**Running a new experiment?** See `evals/contributing.md` for the step-by-step (run → upload data to HF → record an `F<N>` finding here → merge) and the definition-of-done checklist, so the next person inherits both your result and the data to reproduce it.
 
 Plain-English map:
 - A **battery** is a dataset of test conversations or test questions.
@@ -61,7 +61,7 @@ The system under test is the production tutor (`app/`): the same agent, retrieva
 | Tool behavior | Which tools the tutor called, how often, and whether it re-searched for information it had already seen. | Tool-call events saved in each trace bundle. |
 | Retrieval | Whether the retrieved chunks included the correct course or lesson. `recall@shown` means "was the right source among the results shown to the model?" `MRR` rewards putting the right lesson earlier in the shown results. | Ground truth stored on each real discussion case: `source_key` and `lesson_url`. |
 | Response behavior | Whether the tutor did the correct kind of thing: answer from course content, answer generally, redirect a platform/support issue, or acknowledge feedback. | Cheap code checks for early signals; final behavior accuracy comes from human grades. |
-| Answer quality | Whether the answer covered the required key points and whether graded memory probes passed. | Human pass/fail grading in a blinded workbook. LLM judges are used only after they are validated against human labels. |
+| Answer quality | Whether the answer covered the required key points, passed graded memory probes, and (holistic) whether course staff would approve sending it as a genuinely helpful, learning-oriented reply. A separate **faithfulness** check (every claim grounded in retrieved evidence) is built but parked — see the cap correction below. | Human pass/fail grading in a blinded workbook, or the validated blind subagent judge (same per-item rubric). LLM judges are used only after they are validated against human labels. Holistic and probe quality are complementary — holistic is graded blind to conversation history, so it cannot see a memory failure (F34); read them together. |
 | Memory | Whether the tutor remembered or updated facts from earlier in the session, followed preferences, resolved references like "that thing from earlier," and used stored student profiles. | Session probes and persona checks. |
 
 Cost is reported both as raw tokens and as estimated dollars after cached-token discounts. These can rank presets differently (finding F2).
@@ -98,7 +98,7 @@ Command roles:
 - `report` builds side-by-side tables and token curves from graded runs.
 - `handgrade_workbook` builds and merges a blinded grading workbook, so the grader does not know which preset produced an answer.
 
-Every turn persists a JSON trace bundle, so grading and reporting can re-run offline without touching the API. Runs are resume-safe: re-run the same command after an interruption and completed units are skipped. Each turn has a 10-minute timeout, and LangSmith is off by default. Running the tutor needs `GEMINI_API_KEY` + `COHERE_API_KEY`; the full 4-preset comparison run below cost **≈$330** at corrected pricing (the pre-2026-06-13 price table under-priced `gemini-3.5-flash` ~4.4×; the often-quoted "$73" is a pre-correction figure — see Harness corrections).
+Every turn persists a JSON trace bundle, so grading and reporting can re-run offline without touching the API. Runs are resume-safe: re-run the same command after an interruption and completed units are skipped. Each turn has a 10-minute timeout, and LangSmith is off by default. Running the tutor needs `GEMINI_API_KEY` + `COHERE_API_KEY`; the full 4-preset comparison run below cost **≈$323** at corrected pricing (the pre-2026-06-13 price table under-priced `gemini-3.5-flash` ~4.4×; the often-quoted "$73" is a pre-correction figure — see Harness corrections).
 
 **Setup for collaborators.** Code and docs are in git; the datasets and run results contain real student text and live only in the private HF dataset (`towardsai-tutors/ai-tutor-data` — git is force-pushed to the public prod Space on deploys, so student data never enters git). With an `HF_TOKEN` that can read it:
 
@@ -107,7 +107,7 @@ export HF_TOKEN=hf_...    # token with READ access; the inline cmd does NOT load
 mkdir -p data/eval runs  # runs/ is gitignored, so a fresh clone has no runs/ dir yet
 uv run python -c "from huggingface_hub import snapshot_download as d; d(repo_id='towardsai-tutors/ai-tutor-data', repo_type='dataset', allow_patterns=['eval/**','eval_runs/**'], ignore_patterns=['eval/README.md'], local_dir='.')"
 mv eval/* data/eval/ && mv eval_runs/part_*/* runs/   # restore working paths (part_b, part_c, ...)
-mv eval_runs/slm_compaction/axis_a/* runs/ 2>/dev/null   # SLM Axis-A runs -> runs/axisa_* (F31); Axis-B (F30) stays under eval_runs/slm_compaction/axis_b/, see evals_slm_compaction.md
+mv eval_runs/slm_compaction/axis_a/* runs/ 2>/dev/null   # SLM Axis-A runs -> runs/axisa_* (F31); Axis-B (F30) stays under eval_runs/slm_compaction/axis_b/, see evals/slm_compaction.md
 ```
 
 ## Results — Part B comparison run, 2026-06-12
@@ -153,7 +153,7 @@ Numbered findings; each states what it was tested on. Convention: entries are ne
 
 ### Part C screen (2026-06-15)
 
-11 arms (prod + full_history anchors + 9 variants), each on a fixed subset (24 stratified single-turn + 3 sessions s01/s02/s08), 1 trial, ~660 turns, 0 errors, ≈$166 at correct pricing. Graded by the validated subagent judge. Full table: `runs/c_report/report.md`. Probe-accuracy n=12/arm (1 trial) — treat percentages as coarse rankings, not precise rates; promotion (×3 trials, full batteries) would firm them up.
+11 arms (prod + full_history anchors + 9 variants), each on a fixed subset (24 stratified single-turn + 3 sessions s01/s02/s08), 1 trial, ~660 turns, 0 errors, ≈$163 at correct pricing. Graded by the validated subagent judge. Full table: `runs/c_report/report.md`. Probe-accuracy n=12/arm (1 trial) — treat percentages as coarse rankings, not precise rates; promotion (×3 trials, full batteries) would firm them up.
 
 - **F14 — The LLM judge is validated; Part C quality columns are reportable.** A blind subagent re-grade of the 96 human-confirmed probes reproduced them at 98% agreement / TPR 100% / TNR 96% (clears the >90% gate; this measures grader *reproducibility*, since the labels were sign-offs, not blind-from-scratch). The validated grader is the subagent workflow (same per-item-type rubric as `evals/judge.py`), run on subscription at no API cost; all Part C arms are graded that way. Supersedes F13's "judge not yet validated" caveat. (`runs/b_report/judge_val/`.)
 - **F15 — The screen reproduces F9/F10 on independent arms.** `full_history` is cheapest on sessions ($0.10/turn) AND best memory (100% probe); every compaction arm is both pricier and weaker. `prompt_compression` also reaches 100% memory — it shrinks message text but drops no facts — yet saves little (highest tokens), so "don't drop content" preserves memory but is not a cost win. (Screen; n thin.)
@@ -177,14 +177,18 @@ Numbered findings; each states what it was tested on. Convention: entries are ne
 
 - **F27 (2026-06-20) — On single-turn, `full_history`'s latency edge is a *tail* effect from `prod` summarizing mid-turn, not a median win; and `total − ttft` cannot de-confound latency.** Re-read of the Part B/C single-turn bundles (no re-run). Paired per question, the `prod` − `full_history` median **TTFT** delta is **+0.43s** (Part B, n=120, 2 trials) / +3.5s (Part C, n=24, 1 trial) and `prod` is slower in only **62/120** cases — a near-tie at the median, *not* the sessions gap (F9's 17s-vs-21–43s is multi-turn and cannot apply with no prior history). The gap lives in the **tail**: `SummarizationMiddleware` fires on **29/120** single-turn `prod` turns (24%) because the agent's *own* retrieval/browse output crosses the 30k trigger **within one turn**; those turns run **52s TTFT / 16 LLM calls** vs **29s / 8** when it doesn't, and on the *same* questions `prod` does 2–3× `full_history`'s calls (e.g. 9→27, 12→32) — the F9/F4 re-work pathology (summary drops raw evidence → re-browse) inside a single turn. `full_history` never summarizes, so it skips the tail; that lifts its *mean* (Part C 35s vs 51s) while the *median* stays tied. **Methodological corollary (extends the time-of-run caveat below):** `total − ttft` does **not** strip API-load noise — it is the answer-streaming tail (**~6.0s for both presets**; +1.5s even when summarization fires), because the entire agentic loop incl. the summarization tax lands in `ttft`, so subtracting it deletes the signal. Latency ≈ work × time-per-work, and the confound smears across ~20 calls, so no algebra on one sequential run removes it; the **unconfounded** read is the work metrics — **LLM calls/turn and tokens/turn** (run-time-independent like cost/recall, already in every bundle) — which are also the upstream cause. Live LangSmith spot-check (3 interleaved pairs, one browse-heavy single-turn Q) reproduced **both** directions: `prod` faster once (summary collapsed 268k→23k ctx, 65s) and slower twice (re-work: 23/32 calls, 116s/124s, $2.27/$1.04 vs `full_history` $0.74/$0.33), and LangSmith's independent latency matched telemetry to the ms (`prod` 65.07s vs 65.16s; `full_history` 103.64s vs 104.28s), trees confirming `full_history` carries no `SummarizationMiddleware`. Caveats: thin n and time-of-run confounded for absolute seconds; the live pairs are n=3 on one non-representative question. Same class as F24 (single-turn *clearing* backfire) on the other compaction half.
 
-- **F28 (2026-06-19) — GraphRAG does not beat classical hybrid RAG on single-turn course Q&A: it ties on grounding accuracy and costs +44% $/turn, empirically confirming the earlier decision to drop it.** A scoped, fair head-to-head where the *only* variable is the retrieval backend behind `retrieve_tutor_context`: production hybrid `LocalChromaRetriever` (dense Cohere `embed-v4` + BM25 → RRF → Cohere rerank → token budget) vs a true Microsoft GraphRAG index (`app/graph_rag.py` `GraphRAGRetriever` — local-search-style context assembly over entities/text-units/community-reports, mapped back to real `source`/`url` via `corpus_manifest`, then the **same Cohere rerank + token budget** as classical; context-provider only, so the agent's **Gemini 3.5 Flash is the sole generation model in both arms**). 41 `full_stack_ai_engineering` single-turn cases (27 `answer_from_corpus`), Gemini 3.5 Flash, `--disable-kb` so the retriever is the only variable, `--scope-sources`, 1 trial, **0 errors/arm**. Both arms **surface and cite the right source 100%** of the time and tie on lesson recall (**76%**) and cited-correct lesson (**85%**); classical is slightly *better* at ranking the right lesson (**MRR 0.70 vs 0.65**). GraphRAG pulls community-report context, so it spends **+61% input tokens (178k vs 110k) and +44% $/turn ($0.212 vs $0.147)** and is a touch slower, with no accuracy payoff (the +3pt behavior proxy is a coarse n=37 code check, not a quality signal). So GraphRAG's community structure does not help typical single-hop course Q&A — this **empirically supports** the prior "dropped on faith" decision (see "Deliberately not pursued" below) rather than asserting it. **Reusable artifact:** the scoped index (90 docs → 15,059 entities / 32,748 relationships / 3,371 community reports, built with Gemini 2.5 Flash for **$44.96**) is published to the private HF dataset (`graphrag/output/`, pulled on demand, *not* by prod cold-start — `ensure_local_vector_db` ignores `graphrag/**`), so the eval re-runs without the ~$45 rebuild. Full methodology, build/run commands, and the indexing gotchas (route Gemini 2.5 via its OpenAI-compatible endpoint; `run_graphrag_index.sh` loops `graphrag index` until `entities.parquet` exists since one exhausted retry aborts the stage) live in **`evals_graphrag.md`**. Connects to F17/F23 (the `kb_off` grounding discussion — classical RAG already grounds well). Caveats: scoped to one source, single-turn, n=41 (27 corpus-answer), 1 trial, auto metrics only (key-point/behavior human grades pending); `recall@source` is saturated by `--scope-sources`, so lesson recall / MRR / cited-correct-lesson are the discriminating numbers. GraphRAG's theoretical edge is **multi-hop / cross-document synthesis**, which this single-hop battery barely exercises — so this says it doesn't help on single-hop course Q&A, *not* that it never helps; a fair test of that strength needs a multi-hop `_v2` probe set (out of scope here). (Branch `experiment/graphrag-vs-rag`, PR #2.)
+- **F28 (2026-06-19) — GraphRAG does not beat classical hybrid RAG on single-turn course Q&A: it ties on grounding accuracy and costs +44% $/turn, empirically confirming the earlier decision to drop it.** A scoped, fair head-to-head where the *only* variable is the retrieval backend behind `retrieve_tutor_context`: production hybrid `LocalChromaRetriever` (dense Cohere `embed-v4` + BM25 → RRF → Cohere rerank → token budget) vs a true Microsoft GraphRAG index (`app/graph_rag.py` `GraphRAGRetriever` — local-search-style context assembly over entities/text-units/community-reports, mapped back to real `source`/`url` via `corpus_manifest`, then the **same Cohere rerank + token budget** as classical; context-provider only, so the agent's **Gemini 3.5 Flash is the sole generation model in both arms**). 41 `full_stack_ai_engineering` single-turn cases (27 `answer_from_corpus`), Gemini 3.5 Flash, `--disable-kb` so the retriever is the only variable, `--scope-sources`, 1 trial, **0 errors/arm**. Both arms **surface and cite the right source 100%** of the time and tie on lesson recall (**76%**) and cited-correct lesson (**85%**); classical is slightly *better* at ranking the right lesson (**MRR 0.70 vs 0.65**). GraphRAG pulls community-report context, so it spends **+61% input tokens (178k vs 110k) and +44% $/turn ($0.212 vs $0.147)** and is a touch slower, with no accuracy payoff (the +3pt behavior proxy is a coarse n=37 code check, not a quality signal). So GraphRAG's community structure does not help typical single-hop course Q&A — this **empirically supports** the prior "dropped on faith" decision (see "Deliberately not pursued" below) rather than asserting it. **Reusable artifact:** the scoped index (90 docs → 15,059 entities / 32,748 relationships / 3,371 community reports, built with Gemini 2.5 Flash for **$44.96**) is published to the private HF dataset (`graphrag/output/`, pulled on demand, *not* by prod cold-start — `ensure_local_vector_db` ignores `graphrag/**`), so the eval re-runs without the ~$45 rebuild. Full methodology, build/run commands, and the indexing gotchas (route Gemini 2.5 via its OpenAI-compatible endpoint; `run_graphrag_index.sh` loops `graphrag index` until `entities.parquet` exists since one exhausted retry aborts the stage) live in **`evals/graphrag.md`**. Connects to F17/F23 (the `kb_off` grounding discussion — classical RAG already grounds well). Caveats: scoped to one source, single-turn, n=41 (27 corpus-answer), 1 trial, auto metrics only (key-point/behavior human grades pending); `recall@source` is saturated by `--scope-sources`, so lesson recall / MRR / cited-correct-lesson are the discriminating numbers. GraphRAG's theoretical edge is **multi-hop / cross-document synthesis**, which this single-hop battery barely exercises — so this says it doesn't help on single-hop course Q&A, *not* that it never helps; a fair test of that strength needs a multi-hop `_v2` probe set (out of scope here). (Branch `experiment/graphrag-vs-rag`, PR #2.)
 
-- **F29 (2026-06-20) — Keeping a large *static document* in context is a different regime from conversational `full_history`, and it is one boundary where "keep everything" stops winning: for a single long lesson queried repeatedly, retrieving the relevant slice matches keep-it-all at ~1/13 the cost. (Boundary on F9; standalone Gemini-2.5-Flash fleet — absolute numbers not cross-comparable.)** A keep-vs-compact-vs-retrieve study on the corpus's largest lesson (~37.5k tok) loaded once in turn 0, then **15 question-turns with no tools** (the only context is what each method keeps or builds), each answer judged against the full lesson; 11 arms, Gemini 2.5 Flash, 1 trial. `rag` (fetch the relevant chunk per question) is **cheapest *and* tied-best** (60% / 3.2k tok / $0.020); `full_history` (keep the whole lesson in the prompt) is the **priciest in-context arm and not the best** (53% / 43k tok / $0.134); the summarization family cuts ~35% of tokens for a quality drop (40–53%); `incontext_history_retrieval` ties `rag` on quality but at ~13× the tokens; **`hierarchical_summarization` is the worst trade** — priciest ($0.605) and slowest (41.8s p50) for the lowest quality (33%). `graphrag` (53%, $0.045) again does not beat plain `rag` (consistent with **F28**). **Read it as a lesson, not a head-to-head:** this is *not* "RAG beats the `full_history` memory strategy" — here `full_history` only means "the whole lesson sits in the prompt," not its multi-turn-memory role. It says **F9's "keep everything wins" is about *conversation history*; for a big *static document* you could retrieve from, hoarding it in context is the priciest option and buys nothing**, and in the production tutor retrieval (Axis B) and a memory policy (Axis A) **compose**, they don't compete. Companion boundary: F30 (small context window). New reusable presets landed in `app/memory_presets.py`: `delta_summarization`, `hierarchical_summarization`. Full matrix, the F-C1–F-C6 detail, and reproduce steps in **`evals_compaction.md`**. Caveats: n=15, 1 session, 1 trial (coarse rankings, not precise rates); judge is Gemini 2.5 Flash (same family, reads the full lesson as ground truth); **separate fleet from the Part B/C 3.5-Flash runs — quality is not cross-comparable across models**. (Branch `experiment/context-compaction`, PR #5.)
+- **F29 (2026-06-20) — Keeping a large *static document* in context is a different regime from conversational `full_history`, and it is one boundary where "keep everything" stops winning: for a single long lesson queried repeatedly, retrieving the relevant slice matches keep-it-all at ~1/13 the cost. (Boundary on F9; standalone Gemini-2.5-Flash fleet — absolute numbers not cross-comparable.)** A keep-vs-compact-vs-retrieve study on the corpus's largest lesson (~37.5k tok) loaded once in turn 0, then **15 question-turns with no tools** (the only context is what each method keeps or builds), each answer judged against the full lesson; 11 arms, Gemini 2.5 Flash, 1 trial. `rag` (fetch the relevant chunk per question) is **cheapest *and* tied-best** (60% / 3.2k tok / $0.020); `full_history` (keep the whole lesson in the prompt) is the **priciest in-context arm and not the best** (53% / 43k tok / $0.134); the summarization family cuts ~35% of tokens for a quality drop (40–53%); `incontext_history_retrieval` ties `rag` on quality but at ~13× the tokens; **`hierarchical_summarization` is the worst trade** — priciest ($0.605) and slowest (41.8s p50) for the lowest quality (33%). `graphrag` (53%, $0.045) again does not beat plain `rag` (consistent with **F28**). **Read it as a lesson, not a head-to-head:** this is *not* "RAG beats the `full_history` memory strategy" — here `full_history` only means "the whole lesson sits in the prompt," not its multi-turn-memory role. It says **F9's "keep everything wins" is about *conversation history*; for a big *static document* you could retrieve from, hoarding it in context is the priciest option and buys nothing**, and in the production tutor retrieval (Axis B) and a memory policy (Axis A) **compose**, they don't compete. Companion boundary: F30 (small context window). New reusable presets landed in `app/memory_presets.py`: `delta_summarization`, `hierarchical_summarization`. Full matrix, the F-C1–F-C6 detail, and reproduce steps in **`evals/compaction.md`**. Caveats: n=15, 1 session, 1 trial (coarse rankings, not precise rates); judge is Gemini 2.5 Flash (same family, reads the full lesson as ground truth); **separate fleet from the Part B/C 3.5-Flash runs — quality is not cross-comparable across models**. (Branch `experiment/context-compaction`, PR #5.)
 
-- **F30 (2026-06-20) — A small context window is a second boundary where "keep everything" stops winning, and a harder one than F29: when the document physically does not fit, "shove it all" (`full_context`) is forced into truncation and is strictly *dominated*, so retrieving the relevant slice (`rag`) is not just cheaper but necessary. (Small-window companion to F29, Axis B; local-SLM fleet — absolute numbers not cross-comparable.)** "Fit one ~37.7k-token lesson into context to answer 15 questions," 7 methods × 3 Ollama SLMs (llama3.1:8b, qwen2.5:7b-instruct, qwen3:8b no-think) on an M1 Pro/16 GB at a 32k window (`--num-ctx 32768`), no prompt caching; judge stays on Gemini 2.5 Flash (reads the full lesson as ground truth — the SLM cannot hold it, and a model must never grade itself), 1 trial. `full_context` (the whole *document* placed in the prompt, `ctx['full_context'] = lesson`, stateless per question — this is **document-stuffing**, *not* the conversational `full_history` memory policy) is **truncated on 15/15 turns** (37.7k → 32,767) and **dominated**: equal-or-worse quality (67–80%) at **~12–15× the latency** (266–346s vs ~20s) — silent information loss *plus* a huge latency tax (e.g. qwen2.5 `full_context` answered "no model was mentioned" for a fact the truncated intro had named). `rag` (chunk + Cohere-embed + retrieve top-k per question) is **top or tied-top on all three** (qwen2.5 100%, qwen3 100%, llama 80%) at ~2.9k context tokens and ~20–26s. `graphrag` ties `rag` on quality but at ~2.8× tokens and ~2.5× latency — no payoff, **consistent with F28/F29**. Summarization is the floor (llama `summary` **0%**); `trim`/`selective` land mid (47–73%). This marks where F9/F15's precondition — a window big enough to hold everything, cheaply, under caching — simply fails on the hardware a workshop attendee runs. **Distinct from the conversational-memory inversion (F31):** `full_context` here is *document-stuffing* (Axis B), not `full_history`'s multi-turn role, exactly the distinction F29 drew. **Methodology fix:** the first pass scored ~64% of judge verdicts as fails because a long JSON `reason` hit `max_tokens` and truncated the verdict; hardened the judge (25-word reason cap + leading-boolean recovery) and re-graded the saved answers offline (`evals/rejudge_compaction.py`, **0 unparseable** after) — the directly-measured axes (tokens, latency, overflow) were never affected. Caveats: n=15/method, 1 trial, one lesson, one domain — read as a ranking, not rates; **separate fleet (local Ollama SLMs, no caching, Gemini-2.5-Flash judge) — quality/cost not cross-comparable** with the Part B/C 3.5-Flash, the F29 2.5-Flash, or the DeepSeek runs. Full writeup: **`evals_slm_compaction.md`** (Experiment 1). (Branch `experiment/slm-compaction`, PR #3.)
-- **F31 (2026-06-20) — On an SLM, compacting a *growing conversation history* has no universal best method, and "keep everything" (`full_history`) never wins — and model capability dominates the choice of method. (SLM counterpart to F9/F15 on Axis A; same local fleet — absolute numbers not cross-comparable.)** Same lesson loaded turn 0 of a session, 15 question-turns, **retrieval off**, run through the real app middlewares (the production memory presets) on the 3 SLMs at `num_ctx=32768`, judged on Gemini against the full lesson, 1 trial. The best preset is **model-dependent** — `prompt_compression` on qwen2.5 (60%), `hierarchical_summarization` on llama3.1 (40%), `summarization_only`/`delta_summarization` on qwen3 (73%) — so there is **no single "best compaction" to recommend**. `full_history` (the genuine conversational memory policy — keep *every prior turn*, no summarization or context-editing; `summarization=False, context_editing=False`) **tops no model** (27% / 7% / 67%) and is near-bottom on the two weaker ones; `sliding_window` is reliably among the worst (13% / 13% / 40%). **Model capability dominates the method:** qwen3 (40–73% across *every* preset) >> qwen2.5 (13–60%) >> llama3.1 (0–40%) — choosing a stronger small model buys more than choosing the best strategy. **Mechanism (why keep-all can't win here):** on an SLM "keep everything" (`full_history`) can't even be truncated-and-kept — Ollama's server-side context fitting **evicts the whole oversized turn-0 lesson** from turn 1 on (the checkpoint still holds it; the model receives only ~0.3–1.4k tok of accumulated Q&A), so a method that *summarizes the lesson into a small injected message* is the only way to retain its gist; on Gemini's large window this never happens (F9's `full_history` held ~43k/turn). `hierarchical_summarization` is by far the priciest (≈30-call map-reduce, p50 233–333s, ~25–29k retained tok) for no quality lead except on the weakest model. This **inverts F9/F15's conversational-memory result** (keep-all cheapest *and* best) in the regime where the runtime physically cannot keep it. Caveats: n=15, 1 trial, LLM-judge variance moves individual cells ±1–3/15 (read only the big gaps); the `full_history` numbers are **Ollama-eviction-specific** (a different serving layer could keep a truncated lesson instead); **same separate-fleet comparability caveat as F30**. Writeup: **`evals_slm_compaction.md`** (Experiment 2). (Branch `experiment/slm-compaction`, PR #3.)
+- **F30 (2026-06-20) — A small context window is a second boundary where "keep everything" stops winning, and a harder one than F29: when the document physically does not fit, "shove it all" (`full_context`) is forced into truncation and is strictly *dominated*, so retrieving the relevant slice (`rag`) is not just cheaper but necessary. (Small-window companion to F29, Axis B; local-SLM fleet — absolute numbers not cross-comparable.)** "Fit one ~37.7k-token lesson into context to answer 15 questions," 7 methods × 3 Ollama SLMs (llama3.1:8b, qwen2.5:7b-instruct, qwen3:8b no-think) on an M1 Pro/16 GB at a 32k window (`--num-ctx 32768`), no prompt caching; judge stays on Gemini 2.5 Flash (reads the full lesson as ground truth — the SLM cannot hold it, and a model must never grade itself), 1 trial. `full_context` (the whole *document* placed in the prompt, `ctx['full_context'] = lesson`, stateless per question — this is **document-stuffing**, *not* the conversational `full_history` memory policy) is **truncated on 15/15 turns** (37.7k → 32,767) and **dominated**: equal-or-worse quality (67–80%) at **~12–15× the latency** (266–346s vs ~20s) — silent information loss *plus* a huge latency tax (e.g. qwen2.5 `full_context` answered "no model was mentioned" for a fact the truncated intro had named). `rag` (chunk + Cohere-embed + retrieve top-k per question) is **top or tied-top on all three** (qwen2.5 100%, qwen3 100%, llama 80%) at ~2.9k context tokens and ~20–26s. `graphrag` ties `rag` on quality but at ~2.8× tokens and ~2.5× latency — no payoff, **consistent with F28/F29**. Summarization is the floor (llama `summary` **0%**); `trim`/`selective` land mid (47–73%). This marks where F9/F15's precondition — a window big enough to hold everything, cheaply, under caching — simply fails on the hardware a workshop attendee runs. **Distinct from the conversational-memory inversion (F31):** `full_context` here is *document-stuffing* (Axis B), not `full_history`'s multi-turn role, exactly the distinction F29 drew. **Methodology fix:** the first pass scored ~64% of judge verdicts as fails because a long JSON `reason` hit `max_tokens` and truncated the verdict; hardened the judge (25-word reason cap + leading-boolean recovery) and re-graded the saved answers offline (`evals/rejudge_compaction.py`, **0 unparseable** after) — the directly-measured axes (tokens, latency, overflow) were never affected. Caveats: n=15/method, 1 trial, one lesson, one domain — read as a ranking, not rates; **separate fleet (local Ollama SLMs, no caching, Gemini-2.5-Flash judge) — quality/cost not cross-comparable** with the Part B/C 3.5-Flash, the F29 2.5-Flash, or the DeepSeek runs. Full writeup: **`evals/slm_compaction.md`** (Experiment 1). (Branch `experiment/slm-compaction`, PR #3.)
+- **F31 (2026-06-20) — On an SLM, compacting a *growing conversation history* has no universal best method, and "keep everything" (`full_history`) never wins — and model capability dominates the choice of method. (SLM counterpart to F9/F15 on Axis A; same local fleet — absolute numbers not cross-comparable.)** Same lesson loaded turn 0 of a session, 15 question-turns, **retrieval off**, run through the real app middlewares (the production memory presets) on the 3 SLMs at `num_ctx=32768`, judged on Gemini against the full lesson, 1 trial. The best preset is **model-dependent** — `prompt_compression` on qwen2.5 (60%), `hierarchical_summarization` on llama3.1 (40%), `summarization_only`/`delta_summarization` on qwen3 (73%) — so there is **no single "best compaction" to recommend**. `full_history` (the genuine conversational memory policy — keep *every prior turn*, no summarization or context-editing; `summarization=False, context_editing=False`) **tops no model** (27% / 7% / 67%) and is near-bottom on the two weaker ones; `sliding_window` is reliably among the worst (13% / 13% / 40%). **Model capability dominates the method:** qwen3 (40–73% across *every* preset) >> qwen2.5 (13–60%) >> llama3.1 (0–40%) — choosing a stronger small model buys more than choosing the best strategy. **Mechanism (why keep-all can't win here):** on an SLM "keep everything" (`full_history`) can't even be truncated-and-kept — Ollama's server-side context fitting **evicts the whole oversized turn-0 lesson** from turn 1 on (the checkpoint still holds it; the model receives only ~0.3–1.4k tok of accumulated Q&A), so a method that *summarizes the lesson into a small injected message* is the only way to retain its gist; on Gemini's large window this never happens (F9's `full_history` held ~43k/turn). `hierarchical_summarization` is by far the priciest (≈30-call map-reduce, p50 233–333s, ~25–29k retained tok) for no quality lead except on the weakest model. This **inverts F9/F15's conversational-memory result** (keep-all cheapest *and* best) in the regime where the runtime physically cannot keep it. Caveats: n=15, 1 trial, LLM-judge variance moves individual cells ±1–3/15 (read only the big gaps); the `full_history` numbers are **Ollama-eviction-specific** (a different serving layer could keep a truncated lesson instead); **same separate-fleet comparability caveat as F30**. Writeup: **`evals/slm_compaction.md`** (Experiment 2). (Branch `experiment/slm-compaction`, PR #3.)
 - **F32 (2026-06-22) — Synthesis: "keep everything in context" wins only under a specific precondition — a large window that holds the whole history cheaply under prompt caching — and we have now mapped three boundaries where that precondition breaks and keep-all stops winning.** **Two distinct "keep-all" arms — never conflate them:** `full_history` (Axis A — F9/F15/F31) is the *memory policy* that keeps every prior **conversation turn** with no compaction (`summarization=False, context_editing=False` in `app/memory_presets.py`); `full_context` (Axis B — F30) is **document-stuffing** — the whole *source document* placed in the prompt (`ctx['full_context'] = lesson`). Same slogan ("keep everything"), different mechanisms. **F29 is the trap to watch:** its keep-all arm ran the `full_history` *preset* but over a single static lesson parked in turn 0 (`turns = [lesson] + questions`), so what it actually measured is the `full_context` idea (a document held in the prompt), **not** `full_history`'s multi-turn-memory role — which is why F29 sits on the *document* boundary (1) below, alongside F30, and **not** the conversational boundary (3). F9/F15 established the default (raw `full_history` is cheapest *and* best memory on conversational sessions), and F2/F25/F26 explain *why* it holds and even strengthens with horizon (the big **cached** prefix is cheaper than any compaction's cache-breaking rewrite; DeepSeek's ~50× cache pushes the crossover past 36 turns). Remove a piece of that precondition and keep-all loses, along three boundaries: **(1) a large static document you could retrieve from** (F29, Axis B, large cached window) — retrieving the slice *matches* keep-all at ~1/13 the cost, so hoarding the doc is the priciest option and buys nothing; **(2) a context window too small to hold the document** (F30, Axis B, SLM) — keep-all (`full_context`) is forced into truncation and is strictly dominated, so retrieval is *necessary*, not merely cheaper; **(3) a growing history on a small window** (F31, Axis A, SLM) — the runtime *evicts* the oversized turn, so keep-all (`full_history`) never tops the table and some compaction is mandatory, though no single method wins. Two cross-cutting lessons fall out: **model capability dominates the compaction method** on weak local models (F30/F31 — a better small model beats a better strategy), and the **axes compose, they don't compete** (F29) — retrieval (Axis B, *how to fit a document*) and a memory policy (Axis A, *how to compact history*) are orthogonal, so "RAG vs `full_history`" is a category error. **Net guidance:** keep-everything stays the right default in the production regime (large cached cloud window + conversational history; F9/F15/F25/F26); reach for retrieval or compaction exactly when one precondition breaks — a big static doc, a small window, or no caching. **Comparability:** this ties the *lessons* across fleets (3.5-Flash screen, 2.5-Flash F29, DeepSeek F25/F26, local SLMs F30/F31), **not** their absolute numbers, which are not cross-comparable.
 
+- **F33 (2026-06-23) — New quality axis (holistic "would staff send this?"): on single-turn, compaction degrades *answer quality itself*, and the degradation dose-responds to how often summarization fires *within a single turn*. (Independent quality-axis confirmation of F4/F19/F27.)** Added two judge rubrics in `evals/judge.py`: **holistic** (whole-answer staff-approval / does-it-help-the-student-learn gate, emitted for every answered turn) and **faithfulness** (groundedness vs retrieved evidence — *parked*, see the correction below). Graded blind by the validated subagent path (F14) across all single-turn runs; holistic rows added to `evals/report.py`, tagged `[judge]`. The single-turn premise is that memory presets should barely differ ("Memory presets should score about the same here; big differences suggest a bug", §The dataset) — yet holistic spans **57%→100%**, and it is *not* a bug: it tracks within-turn summarization. Dose-response: arms that never summarize mid-turn score 92–100% (`full_history` 92% Part C / 92% Part B n=120; `kb_off` **100%** at just 3.6 LLM calls; `sliding_window`/`incontext_history_retrieval`/`selective_retention`/`retrieval_budget_30k` 96%), while arms whose *own* retrieval/KB output trips the trigger mid-turn sink in trigger order — `observation_truncation` **71%** (22.8 LLM calls/turn, 12/24 turns summarized), `context_reset` **62%** (18/24), `aggressive` **57%** (Part B n=120, **110/120** turns summarized mid-turn). So the F4/F19/F27 within-turn re-work pathology is not merely slower/pricier — it makes the *answers worse*, at a product-relevant magnitude (`aggressive`: ~half its single-turn answers would not be sent). Caveats: Part C n=24 (directional); the firm cell is Part B `aggressive` 57% vs `full_history` 92% at n=120; judge reproducibility-validated on probes (F14), **not** independently on holistic.
+
+- **F34 (2026-06-23) — Across a session, standard compaction's damage is *surgical*: it collapses memory while answer quality stays ~100% — but that "quality" is partly an artifact of a history-blind judge, so "looks fine" ≠ "served the student". Only the most aggressive rewrite (`context_reset`) degrades the answers themselves, progressively over the session. (Refines F10/F12 on the holistic axis.)** Holistic graded on every answered turn of `b_se`+`c_se` (683 turns, 15 arms, blind subagent path). **The decoupling:** `prod` probe **58%**(c)/**38%**(b) yet holistic **97/99%**; `kb_off` probe 33% / holistic 100%; `selective_retention` probe 25% / holistic 100%; `profile_memory` probe 38% / holistic 100% — memory and answer-quality are *independent failure axes*. The lone exception is `context_reset` (probe 17%, holistic **69%**, holistic dropping **−24% early→late turn**; `sliding_window` a milder −16% late): broad, progressive quality degradation appears only under the most destructive history rewriting, otherwise quality holds flat (`Δ late−early` ≈ 0 for all standard arms). **But "high holistic" is misleading on exactly the turns that matter** — holistic is graded *blind to the session history* (sessions carry no per-turn gold answer, and the judge sees only question+answer), so on the **memory-failed probe turns** it still rated the (generic, context-ignoring) answer "good" **100% of the time** for every standard arm (`c_se_prod` 5/5, `b_se_prod` 15/15, `kb_off` 8/8, `selective_retention` 9/9; `forgot+looks-bad` = **0**) — only `context_reset` forgot so badly the blind judge caught it (6/10 forgotten-fact turns also looked bad). The failure mode is therefore *confidently generic*: a well-formed answer that silently ignores what the student stated earlier, which only the targeted probe detects. **Lesson:** holistic (context-free quality) and probe (context-aware memory) measure different things and must be read together — "answer quality looks fine" does **not** mean the student was served. This refines F10 (memory dies but general answer quality survives) and F12 (only the *aggressive* rewrites *also* degrade quality). Caveats: `c_se` is 3 sessions × 1 trial (~18 turns/early-late bucket), `b_se` ×2 trials; session holistic graded without a staff reference (the within-arm early/late comparison controls for that); judge reproducibility-validated on probes (F14), not on holistic — the blind-spot above is itself why a held-out human validation of holistic is the next gate.
+
 **Screen winners → promotion candidates:** `incontext_history_retrieval` (needs the long-session test), `retrieval_budget_30k` and `clear_retrieval_kb` (cheap Axis-B wins; clear_retrieval_kb is cheaper than prod with similar memory + better recall/key-points), anchored by `full_history`. **Drop:** `context_reset`, `observation_truncation`, `selective_retention`.
 
 **Part C — what remains (beyond the workshop).** The screen answered the core question; everything below *hardens* or *extends* it and is **gated on a spend or data decision from Omar** — none is required for the workshop.
@@ -192,7 +196,7 @@ Numbered findings; each states what it was tested on. Convention: entries are ne
 1. **Promotion (rigor, not new findings).** Re-run the 2-3 winners (+ 1-2 combos) on the full batteries (60 single-turn + 32 sessions + 30 replay) × 3 trials, with paired statistical tests and per-variant failure-taxonomy diffs, to turn the screen's coarse n=12 rankings into confident numbers; optionally an Anthropic (Haiku) re-run of the prefix-rewriting arms (`prompt_compression`, `context_reset`, `aggressive`) to test whether explicit caching flips F2's cost ranking. **~$1,300-1,800 at corrected pricing** (the old "$300-400" was at pre-correction prices; × the 4.4 fix — estimate precisely before running, since the per-turn cost is now ~$0.25).
 2. **Finish the v2 contradiction tier — the highest-value new dataset (already built and partially run; see the V2 execution note below).** Moderate sessions: plant A, update to A′, then probe after prod compaction has evicted A from the kept-recent window. This is the one correctness regime where `full_history` itself might lose (it sees both A and A′ and must choose the current one) — yet the partial run so far shows the opposite (`full_history` contradiction **2/2** vs `prod` **0/3**), so that regime has not actually appeared. What remains is finishing Tier 1 (the missing `full_history` session, then Tier-1 `profile_memory` and optionally `incontext_history_retrieval`) and adding trials, not building the dataset. **(2026-06-19 — now run in full on DeepSeek-V4-Flash rather than Gemini: F26 confirms `full_history` 3/3 contradiction vs `prod` 0/3, so the regime did not appear on a second model either; the Gemini Tier-1 finish is now optional cross-model confirmation.)**
 3. **The `incontext_history_retrieval` long-session cost test — principled but less product-critical.** The screen showed `incontext` works (83%, F16) but couldn't show a cost benefit on ≤13-turn sessions. Whether retrieve-old-turns beats `full_history` *once sessions get genuinely long* needs a token-calibrated long-session v2 tier; useful for the workshop argument, but gated by whether that regime matters in real tutor telemetry. **(2026-06-19 — run on DeepSeek (F25) at 36 turns: `full_history` stays cheapest and `incontext` works but does not beat it on cost under first-party caching; whether the answer flips on a 10× cache provider like Gemini/Anthropic is still open.)**
-4. **Conditional builds.** Build `temporal_graph_memory` only if `full_history` fails contradictions (F26: it did **not** on DeepSeek — 3/3 — so this stays unbuilt), `delta_summarization` only if the long-horizon cost tier shows room to beat full history (F25: `full_history` still cheapest at 36t on DeepSeek, so no room there — though a 10× cache provider is untested), and `entity_memory` only if multi-project probes show fact bleed. `collection_memory` is a separate cheap v1/product follow-up for verbatim fact storage (F22 tested the current profile write-back, not that). `hierarchical_summarization` and `sleeptime_consolidation` are deferred. The step-by-step v2 plan is in `evals_part_c_plan.md` → "Battery v2".
+4. **Conditional builds.** Build `temporal_graph_memory` only if `full_history` fails contradictions (F26: it did **not** on DeepSeek — 3/3 — so this stays unbuilt), `delta_summarization` only if the long-horizon cost tier shows room to beat full history (F25: `full_history` still cheapest at 36t on DeepSeek, so no room there — though a 10× cache provider is untested), and `entity_memory` only if multi-project probes show fact bleed. `collection_memory` is a separate cheap v1/product follow-up for verbatim fact storage (F22 tested the current profile write-back, not that). `hierarchical_summarization` and `sleeptime_consolidation` are deferred. The step-by-step v2 plan is in `evals/part_c_plan.md` → "Battery v2".
 
 **Deliberately not pursued.** *Stretch (heavier, likely later):* sub-agent isolation, reframed as a tool-output-token play — keep noisy `run_kb_command` output out of the main context (ties to F1); temporal-graph memory is already the conditional build in item 4 above. *Dropped as low-information given the findings:* the skills / lazy-prompt-loading family (only a ~458-token win — see the talk-outline correction below), procedural memory, GraphRAG (now **empirically tested** rather than dropped on faith — **F28**: ties classical RAG on grounding, costs +44% $/turn, no payoff on single-hop Q&A), and multi-agent / parallel-research agents. Context for the `kb_off` arm (F17/F23): the KB is the dominant grounding tool — ≈89% of Part B turns, ~7.7 KB vs ~0.9 retrieval calls/turn over 1,088 turns — so disabling it forces the 100k-token retrieval fallback, which is why it raised cost/latency.
 
@@ -200,14 +204,16 @@ Numbered findings; each states what it was tested on. Convention: entries are ne
 
 Harness corrections (bugs in our measurement, not findings): the overnight 06-12 stall was machine sleep hanging API streams (all four pipelines stopped the same minute; fixed with the per-turn timeout); battery lesson-URLs carried a `/discussions/` suffix that silently zeroed recall@lesson until normalized (caught by the A3 smoke, re-graded from bundles without re-running).
 
-- **Pricing correction (2026-06-13).** `MODEL_PRICING` under-priced `gemini-3.5-flash` before this date (~$0.30/$2.50 per MTok vs the correct **$1.50 input / $9.00 output / $0.15 cache-read**, verified against Google's price sheet). Every dollar figure generated earlier — the Part B table and the inline costs in findings **F2/F8/F9/F12** — is **~4.4× too low**; token counts are unaffected. Part B bundles were re-costed from their saved token counts (2026-06-15) and `runs/b_report/report.md` regenerated at correct pricing: **relative rankings are unchanged** (F9 still has `full_history` cheapest, $0.11/turn sessions), only absolute dollars move. Real Part A+B spend ≈ **$338**, not ~$73. Use the regenerated report for absolute costs; finding dollars above are pre-correction.
+- **Pricing correction (2026-06-13).** `MODEL_PRICING` under-priced `gemini-3.5-flash` before this date (~$0.30/$2.50 per MTok vs the correct **$1.50 input / $9.00 output / $0.15 cache-read**, verified against Google's price sheet). Every dollar figure generated earlier — the Part B table and the inline costs in findings **F2/F8/F9/F12** — is **~4.4× too low**; token counts are unaffected. Part B bundles were re-costed from their saved token counts (2026-06-15) and `runs/b_report/report.md` regenerated at correct pricing: **relative rankings are unchanged** (F9 still has `full_history` cheapest, $0.11/turn sessions), only absolute dollars move. Real Part A+B spend ≈ **$338**, not ~$73; the comprehensive program ≈ **$590** across all Gemini 3.5 Flash runs (Part B ~$323 · Part C ~$163 · rest ~$103). Use the regenerated report for absolute costs; finding dollars above are pre-correction.
 - **Stable `sheet_row_id` (2026-06-15).** `evals.grade` built `sheet_row_id` with builtin `hash()`, which is salted per process, so regenerating a `handgrade_sheet.csv` produced ids that no longer matched the frozen workbook keymap — silently emptying `handgrade_workbook merge`. Switched to a `hashlib.md5` hash; the Part B re-merge was rebuilt via the deterministic `run_id` to recover the human grades.
 - **`cleared_tool_outputs` is blind to library clearing (2026-06-19).** `ContextEditingMiddleware` edits a per-call `deepcopy` and never persists the placeholder to the checkpoint that `context_window_stats` reads, so the signal is 0 for all presets whether or not `ClearToolUsesEdit` fired (full detail + token evidence in **F24**). Detect clearing via input-token deltas; the custom Part C view-mechanisms avoid this by reporting through the `app.telemetry` turn-signal registry.
+
+- **Faithfulness parked; bundle tool-output cap raised 6k→40k (2026-06-23).** The new **faithfulness** rubric (groundedness vs retrieved evidence, F33) is *not reportable on existing runs*: `evals/run_battery.py` capped each saved tool output at `TOOL_OUTPUT_MAX_CHARS = 6_000` while the KB shell returns up to **40k** (`app/kb_shell.py` `DEFAULT_MAX_OUTPUT_CHARS`), so on KB-heavy arms the judge saw a fraction of what the agent actually grounded on and scored *capture-completeness, not grounding* — faithfulness tracked the KB:retrieval ratio almost monotonically (`b_st_prod` kb:ret 10.3 → **4%**; `c_st_prod` 7.7 → 25%; `kb_off`/`graphrag` kb:ret 0 → 50–76%), the same blind-signal class as **F23/F24**. Raised the cap to **40k** (captures what the agent saw; `output_chars` still records the true length so any overflow stays visible) so *future* re-records make faithfulness measurable; existing bundles cannot be backfilled (content past 6k is gone — needs a re-run). Until then `evals/grade.py` emits faithfulness **only when evidence was captured in full** (`evidence_is_complete`, self-re-enabling after the cap raise), and `evals/report.py` + `evals/grading_merge.py` drop the column (`DROP_ITEM_TYPES`). Holistic (F33/F34) is unaffected — it grades the answer, not the evidence.
 - **Report mislabeled judge grades as human (2026-06-19, fixed).** `evals.report` hardcoded "(human grade)" on the behavior / key-point / probe-accuracy rows, but Part C runs were filled by the LLM judge (`judge_filled.csv`), not a person — so `runs/c_report/report.md` asserted a human label over judge numbers. Fix: `evals.grade` now records a `grade_source` per merged grade (the judge prefixes its note `[judge…]`; humans do not), and `evals.report` tags each column `[judge]`/`[human]`/`[mixed]` and renames the rows to "(graded)". Numbers are unchanged (and judge≈human per F14); only the provenance label is now honest. Same label-vs-measurement class as F23/F24. All Part C arms were re-graded and `c_report` regenerated.
 - **`input tok/turn` is billed-across-calls, not context size (2026-06-19, clarified).** Telemetry sums `usage_metadata` over every internal model call in a turn, so a high-loop arm (`observation_truncation`, 22.8 calls) bills lots of input from small contexts while a big-payload arm (`kb_off`) bills similar totals from few large ones. Correct for *cost*, but it is not "how big is the context window." The report now shows both: **input tok/turn (billed, all calls)** and a new **context tokens/turn (window size)** row (from `context_tokens_approx`, always computed but previously unsurfaced). On single-turn the two even rank arms differently (e.g. `kb_off` has the *largest* window at 73k from its 100k retrieval payloads, but mid-pack billed input). Read F1/F6's "~200k input" as billed volume, not window size.
 - **Latency rows are confounded by time-of-run (caveat, not yet fixed).** `run_part_c_screen.sh` runs arms sequentially, and TTFT/total-ms depend on API load at that moment, so cross-arm latency deltas (e.g. F19's 430s p95, F23's 19s-vs-38s) partly reflect *when* each arm ran. Token/cost/recall are run-time-independent and unaffected. Any latency claim that drives a decision should come from arms run interleaved (or repeated trials), not a single sequential screen. **`total − ttft` does *not* remove this** — it is only the answer-streaming tail (~6s, preset-insensitive); the agentic latency incl. the summarization tax is all in `ttft`, so subtracting it deletes the signal. Report the unconfounded work metrics (LLM calls/turn, tokens/turn) as the headline and treat seconds as supporting. See **F27**.
 - **Latency de-confounding tooling (2026-06-22).** Acting on F27: (1) `runs/run_part_c_screen_interleaved.sh` runs the screen question-outer / arm-inner (writes `runs/ci_*`, not `runs/c_*`) so every question's arms share one API-load window — same total runs as the sequential screen, so interleaving is free. (2) `context_stats` now also emits `time_to_first_token_ms` (first reasoning/tool/text token, always ≤ `ttft_ms`), separating raw model responsiveness from the tool-call loop that `ttft_ms` includes; it flows through `evals/grade.py` to the report and shows `—` on runs recorded before it existed. (3) `evals/report.py` now leads with the run-time-independent work metrics (LLM/tool calls/turn, tokens/turn) and tags every latency row `[confounded]`, with a runtime note. All `runs/*report*` were regenerated (layout/labels only; numbers unchanged). The scripts live in gitignored `runs/` like the rest of the run tooling.
-- **Pattern worth naming — most of our measurement bugs are telemetry/label blind-spots, not wrong math.** Four now share one shape: a number or label asserted a property the instrumentation could not observe for that config — **F21** (mechanism dormant: `profile_memory` never engaged on sessions), **F23** (recall blind to KB browsing), **F24** (clearing invisible to the checkpoint signal), and the judge-as-human label above. `check_triggers` enforces "did the mechanism fire?" for *compaction* only; nothing yet does for retrieval/profile/KB engagement or grade provenance. Standing guard for any new arm: before trusting a per-arm number, confirm the signal/label reflects what the config actually did.
+- **Pattern worth naming — most of our measurement bugs are telemetry/label blind-spots, not wrong math.** Six now share one shape: a number or label asserted a property the instrumentation could not observe for that config — **F21** (mechanism dormant: `profile_memory` never engaged on sessions), **F23** (recall blind to KB browsing), **F24** (clearing invisible to the checkpoint signal), the judge-as-human label above, **faithfulness graded on truncated evidence** (the 6k cap above — the judge scored what it could *see*, not what the agent *saw*), and **F34's holistic-on-session blind spot** (the quality judge sees no conversation history, so it credits answers that silently ignored the student's earlier-stated facts — "looks good" on 100% of memory-failed turns). `check_triggers` enforces "did the mechanism fire?" for *compaction* only; nothing yet does for retrieval/profile/KB engagement, grade provenance, or whether the *grader* saw what it needed. Standing guard for any new arm or metric: before trusting a per-arm number, confirm the signal/label/grader reflects what the config actually did.
 
 Talk-outline correction: the planned "skills / just-in-time KB instructions" change (Change 4) was never built, and its premise is off — the KB instructions block measures **458 tokens** (not ~4k) inside a 1,655-token system prompt; max saving ~2% of a turn, mostly cache-discounted. Recommended reframe: demo progressive disclosure on tool *outputs* (F1), where our numbers actually are.
 
@@ -224,22 +230,22 @@ Talk-outline correction: the planned "skills / just-in-time KB instructions" cha
 - [ ] Next.js meter component over the `data-context-stats` SSE part, if the live demo needs it.
 - [ ] Skills section: reframe per the 458-token measurement above.
 
-**Beyond the workshop.** Part C itself is done and reported above (findings F14–F24); what remains only *hardens* or *extends* it and is consolidated in "Part C — what remains" above — promotion, finishing the v2 contradiction tier, the `incontext_history_retrieval` long-session test, the conditional subsystem builds, and the deliberately-not-pursued list. The full execution detail (variant wiring, telemetry signals, the subset run matrix, the cost model, and the v2 battery plan) lives in `evals_part_c_plan.md`.
+**Beyond the workshop.** Part C itself is done and reported above (findings F14–F24); what remains only *hardens* or *extends* it and is consolidated in "Part C — what remains" above — promotion, finishing the v2 contradiction tier, the `incontext_history_retrieval` long-session test, the conditional subsystem builds, and the deliberately-not-pursued list. The full execution detail (variant wiring, telemetry signals, the subset run matrix, the cost model, and the v2 battery plan) lives in `evals/part_c_plan.md`.
 
 **Product quality track (post-workshop)**
-Golden cases as a CI gate (5–10 critical-path cases with deterministic assertions, including the F7 non-empty check) → error-analysis cycles (an expert reads 50–100 traces in a small viewer, records pass/fail plus the first failure reason, and stops when new failure types stop appearing) → failure taxonomy → code assertions for recurring modes → LLM judges built from the hand-grade critiques and validated to >90% true-positive and true-negative rates on held-out human labels before any judge-graded number is reported → nightly battery + weekly trace sampling. Full methodology behind each step: `evals_background.md`.
+Golden cases as a CI gate (5–10 critical-path cases with deterministic assertions, including the F7 non-empty check) → error-analysis cycles (an expert reads 50–100 traces in a small viewer, records pass/fail plus the first failure reason, and stops when new failure types stop appearing) → failure taxonomy → code assertions for recurring modes → LLM judges built from the hand-grade critiques and validated to >90% true-positive and true-negative rates on held-out human labels before any judge-graded number is reported → nightly battery + weekly trace sampling. Full methodology behind each step: `evals/background.md`.
 
 ## Files
 
 - `evals.md` — this file: the what, the data, the results, the queue.
-- `evals_contributing.md` — contributor playbook: how to run an experiment and land it (run → upload data to HF → record an `F<N>` finding → merge), with a definition-of-done gate.
-- `evals_part_c_plan.md` — Part C execution plan: orchestration (workflow vs direct), the variant catalog with exact wiring + telemetry signals, the subset run matrix, and cost.
-- `evals_background.md` — research sources (Hamel, howtoeval, OpenAI macro-evals) and design rationale.
-- `evals_graphrag.md` — GraphRAG vs classical RAG experiment (branch `experiment/graphrag-vs-rag`): a scoped, true-GraphRAG head-to-head on the single-turn battery. Revisits the dropped GraphRAG idea with a fair test.
-- `evals_compaction.md` — compaction-vs-keep-everything study (branch `experiment/context-compaction`): on the largest lesson, keep-all vs every compaction method vs retrieve-per-question. **Standalone fleet on Gemini 2.5 Flash — not comparable to the 3.5-Flash results above.**
-- `evals_slm_compaction.md` — knowledge-compaction on small local models (branch `experiment/slm-compaction`): two experiments on 3 Ollama SLMs (llama3.1:8b, qwen2.5:7b, qwen3:8b) at a 32k window where the lesson overflows. **Axis B** (fit a document: RAG vs stuff vs summarize, F30) and **Axis A** (compact growing history via the real memory presets, F31). The regime where compaction is *forced*; RAG wins Axis B, no universal winner on Axis A, and "keep everything" never wins on an SLM (synthesis in F32).
+- `evals/contributing.md` — contributor playbook: how to run an experiment and land it (run → upload data to HF → record an `F<N>` finding → merge), with a definition-of-done gate.
+- `evals/part_c_plan.md` — Part C execution plan: orchestration (workflow vs direct), the variant catalog with exact wiring + telemetry signals, the subset run matrix, and cost.
+- `evals/background.md` — research sources (Hamel, howtoeval, OpenAI macro-evals) and design rationale.
+- `evals/graphrag.md` — GraphRAG vs classical RAG experiment (branch `experiment/graphrag-vs-rag`): a scoped, true-GraphRAG head-to-head on the single-turn battery. Revisits the dropped GraphRAG idea with a fair test.
+- `evals/compaction.md` — compaction-vs-keep-everything study (branch `experiment/context-compaction`): on the largest lesson, keep-all vs every compaction method vs retrieve-per-question. **Standalone fleet on Gemini 2.5 Flash — not comparable to the 3.5-Flash results above.**
+- `evals/slm_compaction.md` — knowledge-compaction on small local models (branch `experiment/slm-compaction`): two experiments on 3 Ollama SLMs (llama3.1:8b, qwen2.5:7b, qwen3:8b) at a 32k window where the lesson overflows. **Axis B** (fit a document: RAG vs stuff vs summarize, F30) and **Axis A** (compact growing history via the real memory presets, F31). The regime where compaction is *forced*; RAG wins Axis B, no universal winner on Axis A, and "keep everything" never wins on an SLM (synthesis in F32).
 - `data/eval/README.md` — battery schemas + glossary of terms; `review_log_v1.md` — dataset audit trail.
 - `evals/` — the harness code; `app/memory_presets.py`, `app/telemetry.py` — the app-side hooks.
 - `runs/b_report/`, `runs/c_report/` — generated tables, token curves, blinded grading workbook (Part B and the Part C screen). Re-grade / follow-up reports: `runs/kbfair_report/` (F23 KB-fair recall), `runs/d_report_profile_memory_active/` (F22 profile rerun), `runs/e2_v2_partial_report/` (v2 partial).
 - `runs/ds_v2_t1_report/`, `runs/ds_v2_t2_report/` — **DeepSeek-V4-Flash** v2 matrix (F25/F26: cost + contradiction + long-horizon), graded via the subagent path (`runs/ds_judge/`). Shared on HF as `eval_runs/part_f_deepseek_v2/` (the standard `mv eval_runs/part_*/* runs/` download restores them). Provider added in `app/chat_service.py` (`deepseek` / `openrouter`), pricing in `app/telemetry.py`.
-- `runs/axisa_*_report/`, `data/compaction_slm_*/` — **SLM compaction** study (F30/F31/F32, branch `experiment/slm-compaction`): Axis B fit-a-document and Axis A compact-history on 3 Ollama SLMs at a 32k window. Shared on HF under `eval_runs/slm_compaction/` (`axis_b/<model>/`, `axis_a/axisa_<model>_<preset>/`) + batteries `eval/compaction/`; the snippet above restores Axis-A to `runs/axisa_*`, Axis-B stays under `eval_runs/slm_compaction/axis_b/` — full restore command in `evals_slm_compaction.md`. Provider `ollama` added in `app/chat_service.py`.
+- `runs/axisa_*_report/`, `data/compaction_slm_*/` — **SLM compaction** study (F30/F31/F32, branch `experiment/slm-compaction`): Axis B fit-a-document and Axis A compact-history on 3 Ollama SLMs at a 32k window. Shared on HF under `eval_runs/slm_compaction/` (`axis_b/<model>/`, `axis_a/axisa_<model>_<preset>/`) + batteries `eval/compaction/`; the snippet above restores Axis-A to `runs/axisa_*`, Axis-B stays under `eval_runs/slm_compaction/axis_b/` — full restore command in `evals/slm_compaction.md`. Provider `ollama` added in `app/chat_service.py`.

evals/grade.py

@@ -50,6 +50,80 @@ RETRIEVAL_TOOLS = {"retrieve_tutor_context", "run_kb_command"}
 # in the command or its output reveal which corpus source the agent touched.
 KB_RAW_SOURCE_RE = re.compile(r"raw/(?:docs|courses)/([A-Za-z0-9_.\-]+)/")
 
+# Per-bundle evidence cap for faithfulness rows. Tool outputs in a bundle are each
+# capped at ~6000 chars; a few calls per turn, so ~12k keeps the gist of every
+# call without bloating the grading prompt.
+FAITHFULNESS_EVIDENCE_MAX = 12_000
+
+# Holistic = the "would staff approve sending this?" gate (evals/background.md
+# Layer 4). Emitted for every single-turn answer; the full rubric lives in
+# evals/judge.py RUBRICS["holistic"], this is the at-a-glance criterion.
+HOLISTIC_CRITERION = (
+    "Would an experienced course tutor approve sending this answer to the "
+    "student? Pass only if it is correct, grounded in the right material, and "
+    "genuinely helps them learn (builds understanding), with appropriate scope "
+    "and tone. (see README / judge rubric)"
+)
+# Faithfulness = is the answer grounded in the evidence the tutor actually
+# retrieved (catches hallucination on corpus questions). Emitted only when the
+# turn produced retrieval/KB evidence; rubric in judge.py RUBRICS["faithfulness"].
+FAITHFULNESS_CRITERION = (
+    "Are the answer's substantive factual/technical claims supported by the "
+    "retrieved evidence shown below? Fail on fabricated APIs, parameters, "
+    "versions, citations, or course/lesson specifics not in the evidence."
+)
+
+
+def faithfulness_evidence(
+    bundle: dict[str, Any], max_chars: int = FAITHFULNESS_EVIDENCE_MAX
+) -> str:
+    """Concatenate the retrieval/KB evidence the tutor saw, for grounding grades.
+
+    Joins each retrieval/KB tool call's command + output (and, for
+    `retrieve_tutor_context`, the ranked source titles/URLs) so the judge can
+    check the answer's claims against exactly what the tools returned. Empty when
+    the turn used no retrieval/KB tool (e.g. answer_general), which is the signal
+    to skip the faithfulness row for that case.
+    """
+    parts: list[str] = []
+    for call in bundle.get("tool_calls") or []:
+        name = call.get("tool_name")
+        if name not in RETRIEVAL_TOOLS:
+            continue
+        header = f"[{name}] {(call.get('args_text') or '').strip()}".strip()
+        body = (call.get("output_text") or "").strip()
+        matches = call.get("matches") or []
+        if matches:
+            cites = "; ".join(
+                f"{m.get('title', '')} <{m.get('url', '')}>" for m in matches[:10]
+            )
+            body = f"{body}\nSOURCES: {cites}".strip()
+        if header or body:
+            parts.append(f"{header}\n{body}".strip())
+    return "\n\n".join(parts)[:max_chars]
+
+
+def evidence_is_complete(bundle: dict[str, Any]) -> bool:
+    """True if every retrieval/KB tool output was captured in full.
+
+    Bundles cap each tool output at ``run_battery.TOOL_OUTPUT_MAX_CHARS`` but
+    preserve the true length in ``output_chars``. When a KB-browse output was
+    truncated, the judge sees only a slice of what the agent actually read, so a
+    faithfulness grade would measure capture-completeness, not grounding (the
+    same blind-signal class as evals.md F23/F24). Faithfulness is therefore
+    emitted ONLY when nothing the agent grounded on was truncated; after the
+    bundle cap is raised and a run is re-recorded, this returns True and the
+    column populates automatically.
+    """
+    for call in bundle.get("tool_calls") or []:
+        if call.get("tool_name") not in RETRIEVAL_TOOLS:
+            continue
+        full = call.get("output_chars")
+        kept = len(call.get("output_text") or "")
+        if isinstance(full, int) and full > kept:
+            return False
+    return True
+
 
 def index_battery(battery: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
     index: dict[str, dict[str, Any]] = {}
@@ -207,6 +281,7 @@ def sheet_row(
     item_type: str,
     criterion: str,
     reference: str = "",
+    reference_max: int = 2000,
 ) -> dict[str, str]:
     return {
         # Stable hash (NOT builtin hash(), which is salted per process — that
@@ -221,12 +296,49 @@ def sheet_row(
         "question": bundle["query"][:600],
         "answer": (bundle.get("answer") or "")[:4000],
         "criterion": criterion,
-        "reference": reference[:2000],
+        "reference": reference[:reference_max],
         "grade": "",
         "note": "",
     }
 
 
+def quality_sheet_rows(
+    bundle: dict[str, Any], reference: str = ""
+) -> list[dict[str, str]]:
+    """Holistic + faithfulness sheet rows for one answered turn.
+
+    Shared by single-turn and session turns: holistic (would staff approve this
+    answer?) is emitted for every answered turn; faithfulness only when the turn
+    produced retrieval/KB evidence to ground against. ``reference`` is the staff
+    answer for single-turn cases (context for holistic) and empty for session
+    turns, which have no per-turn gold answer.
+    """
+    rows = [
+        sheet_row(
+            bundle=bundle,
+            item_type="holistic",
+            criterion=HOLISTIC_CRITERION,
+            reference=reference,
+        )
+    ]
+    # Faithfulness only when the agent's grounding was captured in full -- on
+    # bundles recorded under the old 6k tool-output cap, KB-browse evidence is
+    # truncated, so a faithfulness grade would score capture-completeness, not
+    # grounding (evals.md F23/F24 class). Re-record with the raised cap to enable.
+    evidence = faithfulness_evidence(bundle)
+    if evidence and evidence_is_complete(bundle):
+        rows.append(
+            sheet_row(
+                bundle=bundle,
+                item_type="faithfulness",
+                criterion=FAITHFULNESS_CRITERION,
+                reference=evidence,
+                reference_max=FAITHFULNESS_EVIDENCE_MAX,
+            )
+        )
+    return rows
+
+
 def grade_run(run_dir: Path) -> tuple[list[dict[str, Any]], list[dict[str, str]]]:
     bundles = load_jsonl(run_dir / "bundles.jsonl")
     if not bundles:
@@ -306,6 +418,11 @@ def grade_run(run_dir: Path) -> tuple[list[dict[str, Any]], list[dict[str, str]]
                         reference=record.get("reference_answer") or "",
                     )
                 )
+            # Holistic (whole-answer staff-approval) + faithfulness (grounded in
+            # retrieved evidence); staff answer is context for holistic.
+            sheet.extend(
+                quality_sheet_rows(bundle, record.get("reference_answer") or "")
+            )
 
         elif bundle["battery_type"] == "sessions":
             probe = next(
@@ -332,6 +449,11 @@ def grade_run(run_dir: Path) -> tuple[list[dict[str, Any]], list[dict[str, str]]
                         f"Rule: {probe['check_note']}",
                     )
                 )
+            # Holistic + faithfulness on every answered session turn (no per-turn
+            # gold answer, so no reference for holistic). Lets us see whether
+            # answer quality degrades on late turns under compaction.
+            if bundle.get("answer"):
+                sheet.extend(quality_sheet_rows(bundle))
 
         elif bundle["battery_type"] == "personas":
             result = grade_persona_question(record, bundle.get("answer") or "")
@@ -356,6 +478,11 @@ def grade_run(run_dir: Path) -> tuple[list[dict[str, Any]], list[dict[str, str]]
                         criterion=" | ".join(llm_checks),
                     )
                 )
+            # Holistic + faithfulness on persona answers too (full quality
+            # coverage across batteries); the persona profile is the implicit
+            # reference, so no staff reference text here.
+            if bundle.get("answer"):
+                sheet.extend(quality_sheet_rows(bundle))
 
         elif bundle["battery_type"] == "replay":
             sheet.append(
@@ -391,22 +518,47 @@ def merge_handgrades(
             for r in rows
         }
         grade["grade_source"] = sources.pop() if len(sources) == 1 else "mixed"
+
+        def _src(row: dict[str, str]) -> str:
+            # Per-metric provenance so a run with human key-points but judge-filled
+            # holistic/faithfulness still reports each row's true grader (the
+            # overall grade_source above goes "mixed" in that case).
+            return (
+                "judge"
+                if (row.get("note") or "").lstrip().startswith("[judge")
+                else "human"
+            )
+
         key_points = [r for r in rows if r["item_type"] == "key_point"]
         if key_points:
             passed = sum(1 for r in key_points if r["grade"].lower() == "pass")
             grade["key_points_passed"] = passed
             grade["key_points_total"] = len(key_points)
+            kp_sources = {_src(r) for r in key_points}
+            grade["key_points_source"] = (
+                kp_sources.pop() if len(kp_sources) == 1 else "mixed"
+            )
         for row in rows:
             verdict = row["grade"].strip().lower() == "pass"
             if row["item_type"] == "behavior":
                 grade["behavior_pass"] = verdict
+                grade["behavior_source"] = _src(row)
             elif row["item_type"].startswith("probe:"):
                 grade["probe_pass"] = verdict
+                grade["probe_source"] = _src(row)
             elif row["item_type"] == "persona_llm_check":
                 # Combines with the regex auto result (both must pass).
                 grade["auto_pass"] = verdict and grade.get("auto_pass") is not False
+                grade["persona_source"] = _src(row)
             elif row["item_type"] == "replay_reply":
                 grade["replay_pass"] = verdict
+                grade["replay_source"] = _src(row)
+            elif row["item_type"] == "holistic":
+                grade["holistic_pass"] = verdict
+                grade["holistic_source"] = _src(row)
+            elif row["item_type"] == "faithfulness":
+                grade["faithfulness_pass"] = verdict
+                grade["faithfulness_source"] = _src(row)
     return grades
 
 

evals/grade_workflow.js

@@ -0,0 +1,66 @@
+export const meta = {
+  name: 'subagent-judge-grading',
+  description: 'Blinded subagent judge grades holistic/faithfulness/key-point/behavior chunks',
+  phases: [{ title: 'Grade', detail: 'one blinded grader agent per chunk' }],
+}
+
+// args: { battery, chunks: <int> }  OR  { battery, indices: [<int>...] }
+// indices lets us re-run only the missing chunks after a partial run.
+let A = args
+if (typeof A === 'string') {
+  try { A = JSON.parse(A) } catch { A = {} }
+}
+const battery = (A && A.battery) || 'singleturn'
+const explicit = A && Array.isArray(A.indices) ? A.indices : null
+const nChunks = (A && A.chunks) || 0
+if (!explicit && !nChunks) throw new Error('pass args.chunks (count) or args.indices (array)')
+
+const dir = `runs/_grading/${battery}`
+
+const SUMMARY_SCHEMA = {
+  type: 'object',
+  additionalProperties: false,
+  properties: {
+    chunk: { type: 'integer' },
+    graded: { type: 'integer', description: 'rows graded and written' },
+    low_confidence: { type: 'integer', description: 'count of confidence=low verdicts' },
+    wrote: { type: 'string', description: 'path of the verdicts json written' },
+  },
+  required: ['chunk', 'graded', 'low_confidence', 'wrote'],
+}
+
+function prompt(i) {
+  const pad = String(i).padStart(3, '0')
+  const chunkPath = `${dir}/chunk_${pad}.csv`
+  const outPath = `${dir}/verdicts_${pad}.json`
+  return `You are a BLINDED grader for an AI tutor on a course platform (applied AI, LLMs, RAG, Python).
+
+INPUT: parse the CSV at ${chunkPath} with Python (use csv with field_size_limit(10_000_000); fields contain newlines). Columns: sheet_row_id, item_type, question, criterion, reference, answer.
+
+RUBRICS: read the dict RUBRICS and the string PREAMBLE in evals/judge.py and follow them EXACTLY, selecting the rubric by each row's item_type (for "probe:*" use the "probe" rubric; for "faithfulness" the 'reference' field IS the retrieved evidence to grade grounding against; for unknown types use "key_point"). These are the same rubrics the project's validated judge uses.
+
+RULES:
+- Grade EVERY row in the chunk. You see only content; you do NOT know which system/preset produced each answer and must not infer or use it.
+- An EMPTY answer = fail (confidence high).
+- grade is "pass" or "fail"; always decide. confidence is "high" or "low" — use "low" ONLY for genuinely borderline calls a reasonable grader could see either way. reason = one short sentence.
+
+OUTPUT: write a JSON file to ${outPath} that is a list of objects, one per row, each: {"sheet_row_id": <str>, "item_type": <str>, "grade": "pass"|"fail", "confidence": "high"|"low", "reason": <str>}. Write it with Python json.dump. Every input sheet_row_id must appear exactly once.
+
+Then return the summary (chunk index ${i}, how many you graded, how many were low confidence, and the output path). Your returned text is parsed as data, not shown to a human.`
+}
+
+phase('Grade')
+const indices = explicit || Array.from({ length: nChunks }, (_, i) => i)
+const results = await parallel(
+  indices.map((i) => () =>
+    agent(prompt(i), { schema: SUMMARY_SCHEMA, label: `${battery}:chunk_${i}`, phase: 'Grade' })
+  )
+)
+
+const ok = results.filter(Boolean)
+const graded = ok.reduce((a, r) => a + (r.graded || 0), 0)
+const low = ok.reduce((a, r) => a + (r.low_confidence || 0), 0)
+const failedChunks = indices.filter((_, k) => !results[k])
+log(`${battery}: ${ok.length}/${indices.length} chunks done, ${graded} rows graded, ${low} low-confidence`)
+if (failedChunks.length) log(`FAILED chunks (rerun): ${failedChunks.join(', ')}`)
+return { battery, chunks_done: ok.length, chunks_total: indices.length, graded, low_confidence: low, failed_chunks: failedChunks }

evals/grading_merge.py

@@ -0,0 +1,247 @@
+"""Merge subagent-judge verdicts back into runs, and collect uncertain calls.
+
+Companion to grading_prep.py / grade_workflow.js. For one battery:
+1. load every runs/_grading/<battery>/verdicts_*.json,
+2. per run, write combined_filled.csv = kept existing grades + new judge verdicts,
+   then run `evals.grade --handgrades` to regenerate grades_merged.jsonl,
+3. flag any expected rows that never got a verdict (chunk agent failed), and
+4. write uncertain.md + uncertain.csv = every confidence=low verdict with its
+   question/answer/criterion (the human-review queue the user asked for).
+"""
+
+from __future__ import annotations
+
+import collections
+import csv
+import hashlib
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+csv.field_size_limit(10_000_000)
+
+
+def _ah(answer: str) -> str:
+    """Short hash of the graded answer text, to disambiguate sheet_row_id.
+
+    sheet_row_id = run_id|item_type|md5(criterion) is NOT globally unique: run_id
+    is per-case, so the same question graded under two presets (different run
+    dirs) shares an id while having different answers. Keying verdicts by
+    (sheet_row_id, answer-hash) restores uniqueness; identical answers collapse
+    correctly (same grade).
+    """
+    return hashlib.md5((answer or "").encode("utf-8")).hexdigest()[:12]
+
+
+COMBINED_COLS = [
+    "sheet_row_id",
+    "run_id",
+    "battery_type",
+    "preset",
+    "item_type",
+    "criterion",
+    "grade",
+    "note",
+]
+
+# Item types dropped from the merge + review queue. Faithfulness is parked: on
+# bundles recorded under the old 6k tool-output cap the judge can't see the
+# truncated KB evidence, so it scores capture-completeness not grounding
+# (evals.md F23/F24 class). Re-enable after re-recording runs with the raised cap.
+DROP_ITEM_TYPES = {"faithfulness"}
+
+
+def load_verdicts(
+    gdir: Path,
+) -> tuple[dict[tuple[str, str], dict], dict[tuple[str, str], dict]]:
+    """Return (verdicts, content) keyed by (sheet_row_id, answer-hash).
+
+    Recovers the verdict<->row association by POSITION within each chunk (chunk
+    row i corresponds to verdict i; order verified identical for all chunks), so
+    the per-run answer the verdict actually graded is known and used as the key.
+    `content` carries the chunk row (question/criterion/answer) for the report.
+    """
+    verdicts: dict[tuple[str, str], dict] = {}
+    content: dict[tuple[str, str], dict] = {}
+    for cf in sorted(gdir.glob("chunk_*.csv")):
+        crows = list(csv.DictReader(cf.open()))
+        vf = gdir / cf.name.replace("chunk_", "verdicts_").replace(".csv", ".json")
+        if not vf.exists():
+            continue
+        vrows = json.loads(vf.read_text())
+        # Match by sheet_row_id with per-id FIFO queues: robust to a short chunk
+        # (agent skipped rows), reordering, or within-chunk duplicate ids (a chunk
+        # spanning two runs that share a session/turn). A chunk row with no
+        # remaining verdict is left unmatched -> surfaces as "missing" in main().
+        queues: dict[str, collections.deque] = collections.defaultdict(
+            collections.deque
+        )
+        for v in vrows:
+            queues[v["sheet_row_id"]].append(v)
+        for r in crows:
+            q = queues.get(r["sheet_row_id"])
+            if not q:
+                continue
+            v = q.popleft()
+            key = (r["sheet_row_id"], _ah(r.get("answer") or ""))
+            verdicts[key] = v
+            content[key] = r
+    return verdicts, content
+
+
+def expected_keys(gdir: Path) -> set[tuple[str, str]]:
+    keys: set[tuple[str, str]] = set()
+    for ch in sorted(gdir.glob("chunk_*.csv")):
+        for r in csv.DictReader(ch.open()):
+            keys.add((r["sheet_row_id"], _ah(r.get("answer") or "")))
+    return keys
+
+
+def note_for(v: dict) -> str:
+    return f"[judge:{v.get('confidence', 'high')}] {v.get('reason', '')}".strip()
+
+
+def main() -> None:
+    battery = sys.argv[1] if len(sys.argv) > 1 else "singleturn"
+    gdir = Path("runs/_grading") / battery
+    manifest = json.loads((gdir / "manifest.json").read_text())
+    verdicts, content = load_verdicts(gdir)
+    expected = expected_keys(gdir)
+    missing = sorted(expected - set(verdicts))
+    print(f"{battery}: {len(verdicts)} verdicts, {len(missing)} expected rows missing")
+
+    merged_runs = 0
+    key_to_run: dict[tuple[str, str], str] = {}
+    for name in manifest["runs"]:
+        run_dir = Path("runs") / name
+        sheet = list(csv.DictReader((run_dir / "handgrade_sheet.csv").open()))
+        combined: list[dict] = []
+        # kept existing grades (columns may differ across old files; normalize)
+        keep_path = gdir / "keep" / f"{name}.csv"
+        if keep_path.exists():
+            for r in csv.DictReader(keep_path.open()):
+                combined.append({c: r.get(c, "") for c in COMBINED_COLS})
+        # new judge verdicts for this run's sheet rows (content key disambiguates
+        # the cross-run sheet_row_id collision)
+        for r in sheet:
+            if r["item_type"] in DROP_ITEM_TYPES:
+                continue
+            key = (r["sheet_row_id"], _ah(r.get("answer") or ""))
+            key_to_run[key] = name
+            v = verdicts.get(key)
+            if not v:
+                continue
+            combined.append(
+                {
+                    "sheet_row_id": r["sheet_row_id"],
+                    "run_id": r.get("run_id", ""),
+                    "battery_type": r.get("battery_type", battery),
+                    "preset": r.get("preset", ""),
+                    "item_type": r["item_type"],
+                    "criterion": r.get("criterion", ""),
+                    "grade": v["grade"],
+                    "note": note_for(v),
+                }
+            )
+        cf = run_dir / "combined_filled.csv"
+        with cf.open("w", newline="") as f:
+            w = csv.DictWriter(f, fieldnames=COMBINED_COLS, extrasaction="ignore")
+            w.writeheader()
+            w.writerows(combined)
+        res = subprocess.run(
+            [
+                "uv",
+                "run",
+                "-m",
+                "evals.grade",
+                "--run",
+                str(run_dir),
+                "--handgrades",
+                str(cf),
+            ],
+            capture_output=True,
+            text=True,
+        )
+        if res.returncode != 0:
+            print(f"  MERGE FAILED {name}: {res.stderr.strip()[-300:]}")
+        else:
+            merged_runs += 1
+    print(f"merged {merged_runs}/{len(manifest['runs'])} runs")
+
+    # ---- uncertain queue (the deliverable) --------------------------------
+    low = [
+        (k, v)
+        for k, v in verdicts.items()
+        if v.get("confidence") == "low"
+        and v.get("item_type", "").split(":")[0] not in DROP_ITEM_TYPES
+    ]
+
+    def arm_of(key: tuple[str, str]) -> str:
+        return key_to_run.get(key, "?")
+
+    low.sort(key=lambda kv: (kv[1]["item_type"], arm_of(kv[0])))
+    with (gdir / "uncertain.csv").open("w", newline="") as f:
+        w = csv.DictWriter(
+            f,
+            fieldnames=[
+                "sheet_row_id",
+                "run",
+                "item_type",
+                "judge_grade",
+                "judge_reason",
+                "question",
+                "criterion",
+                "answer",
+                "your_grade",
+            ],
+        )
+        w.writeheader()
+        for key, v in low:
+            c = content.get(key, {})
+            w.writerow(
+                {
+                    "sheet_row_id": key[0],
+                    "run": arm_of(key),
+                    "item_type": v["item_type"],
+                    "judge_grade": v["grade"],
+                    "judge_reason": v.get("reason", ""),
+                    "question": (c.get("question") or "")[:300],
+                    "criterion": (c.get("criterion") or "")[:300],
+                    "answer": (c.get("answer") or "")[:1200],
+                    "your_grade": "",
+                }
+            )
+
+    lines = [f"# Uncertain judge calls — {battery}", ""]
+    lines.append(
+        f"{len(low)} low-confidence verdicts of {len(verdicts)} graded. "
+        "Fill `your_grade` in uncertain.csv to override.\n"
+    )
+    by_type = collections.Counter(v["item_type"].split(":")[0] for _, v in low)
+    by_run = collections.Counter(arm_of(k) for k, _ in low)
+    lines.append(
+        "By item type: " + ", ".join(f"{k}={n}" for k, n in sorted(by_type.items()))
+    )
+    lines.append(
+        "By run: " + ", ".join(f"{k}={n}" for k, n in sorted(by_run.items())) + "\n"
+    )
+    for key, v in low:
+        c = content.get(key, {})
+        lines.append(
+            f"## [{v['item_type']}] judge={v['grade'].upper()} — run `{arm_of(key)}`"
+        )
+        lines.append(f"- **id**: `{key[0]}`")
+        lines.append(f"- **judge reason**: {v.get('reason', '')}")
+        lines.append(f"- **Q**: {(c.get('question') or '')[:300]}")
+        lines.append(f"- **criterion**: {(c.get('criterion') or '')[:300]}")
+        lines.append(f"- **answer**: {(c.get('answer') or '')[:700]}")
+        lines.append("")
+    (gdir / "uncertain.md").write_text("\n".join(lines) + "\n")
+    print(f"uncertain: {len(low)} low-confidence -> {gdir}/uncertain.md (+ .csv)")
+    if missing:
+        print("MISSING keys (rerun their chunks):", len(missing))
+
+
+if __name__ == "__main__":
+    main()

evals/grading_prep.py

@@ -0,0 +1,171 @@
+"""Prep subagent-judge grading inputs from regenerated handgrade sheets.
+
+Not part of the shipped harness; a one-off orchestration helper for the
+holistic/faithfulness backfill (kept in-repo so the run is reproducible).
+
+Per run, per item_type the column is treated as all-or-nothing:
+- already fully graded (existing judge_filled/handgrade_filled covers every sheet
+  row of that type) -> KEEP those grades, do not re-grade;
+- otherwise -> ALL sheet rows of that type go to the judge (a single grader per
+  column per run, so no human+judge double-counting and clean provenance).
+
+Outputs under runs/_grading/<battery>/:
+- chunk_NNN.csv      blinded rows for one grader agent (sheet_row_id + content)
+- keep/<run>.csv     existing grades to carry into the final merge
+- manifest.json      battery -> {n_chunks, n_rows, runs}
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+import sys
+from collections import Counter, defaultdict
+from pathlib import Path
+
+from .grading_merge import DROP_ITEM_TYPES
+
+csv.field_size_limit(10_000_000)
+
+SKIP = {
+    "a4_s03_aggressive",
+    "a4_s03_fullhist",
+    "a4_sessions_prod",
+    "smoke_personas",
+    "smoke_session_prod",
+    "smoke_st_fullhist",
+    "smoke_st_prod",
+    "ds_smoke_st",
+    "e_v2_prod",
+}
+MAX_ROWS = 40
+MAX_CHARS = 300_000
+AGENT_COLS = [
+    "sheet_row_id",
+    "item_type",
+    "question",
+    "criterion",
+    "reference",
+    "answer",
+]
+
+
+def battery_type(run_dir: Path) -> str:
+    first = json.loads((run_dir / "bundles.jsonl").open().readline())
+    return first.get("battery_type")
+
+
+def existing_filled(run_dir: Path) -> list[dict[str, str]]:
+    out: list[dict[str, str]] = []
+    for fn in ("judge_filled.csv", "handgrade_filled.csv"):
+        p = run_dir / fn
+        if p.exists():
+            out += [
+                r for r in csv.DictReader(p.open()) if (r.get("grade") or "").strip()
+            ]
+    return out
+
+
+def plan_run(run_dir: Path) -> tuple[list[dict[str, str]], list[dict[str, str]]]:
+    """Return (rows_to_grade, rows_to_keep) for one run."""
+    sheet = list(csv.DictReader((run_dir / "handgrade_sheet.csv").open()))
+    if not sheet:
+        return [], []
+    sheet_by_type: dict[str, list] = defaultdict(list)
+    for r in sheet:
+        sheet_by_type[r["item_type"]].append(r)
+    filled = existing_filled(run_dir)
+    filled_by_type: dict[str, list] = defaultdict(list)
+    for r in filled:
+        filled_by_type[r["item_type"]].append(r)
+
+    to_grade: list[dict[str, str]] = []
+    to_keep: list[dict[str, str]] = []
+    for itype, rows in sheet_by_type.items():
+        graded = filled_by_type.get(itype, [])
+        if len(graded) >= len(rows):  # column already fully graded -> keep
+            to_keep.extend(graded)
+        else:  # grade the whole column
+            to_grade.extend(rows)
+    return to_grade, to_keep
+
+
+def chunk(rows: list[dict[str, str]]) -> list[list[dict[str, str]]]:
+    chunks: list[list[dict[str, str]]] = []
+    cur: list[dict[str, str]] = []
+    cur_chars = 0
+    for r in rows:
+        rchars = len(r.get("reference") or "") + len(r.get("answer") or "")
+        if cur and (len(cur) >= MAX_ROWS or cur_chars + rchars > MAX_CHARS):
+            chunks.append(cur)
+            cur, cur_chars = [], 0
+        cur.append(r)
+        cur_chars += rchars
+    if cur:
+        chunks.append(cur)
+    return chunks
+
+
+def main() -> None:
+    want = sys.argv[1] if len(sys.argv) > 1 else "singleturn"
+    # Optional 2nd arg: comma-separated run-name prefixes to include (focus a run
+    # on a subset of arms, e.g. "b_se,c_se" for the sessions quality test).
+    prefixes = tuple(
+        p for p in (sys.argv[2].split(",") if len(sys.argv) > 2 else []) if p
+    )
+    base = Path("runs")
+    out = base / "_grading" / want
+    (out / "keep").mkdir(parents=True, exist_ok=True)
+    # answers come from bundles (sheet 'answer' is truncated to 4000; fine for grading)
+    all_to_grade: list[dict[str, str]] = []
+    runs_used: list[str] = []
+    by_type = Counter()
+    for run_dir in sorted(base.glob("*/")):
+        name = run_dir.name
+        if name in SKIP or name.startswith("axisa"):
+            continue
+        if prefixes and not name.startswith(prefixes):
+            continue
+        if not (run_dir / "handgrade_sheet.csv").exists():
+            continue
+        if battery_type(run_dir) != want:
+            continue
+        to_grade, to_keep = plan_run(run_dir)
+        # Faithfulness is parked (see grading_merge.DROP_ITEM_TYPES): don't spend
+        # grading budget on it while the report/merge drop it.
+        to_grade = [r for r in to_grade if r["item_type"] not in DROP_ITEM_TYPES]
+        if to_keep:
+            with (out / "keep" / f"{name}.csv").open("w", newline="") as f:
+                w = csv.DictWriter(f, fieldnames=to_keep[0].keys())
+                w.writeheader()
+                w.writerows(to_keep)
+        if to_grade:
+            runs_used.append(name)
+            all_to_grade.extend(to_grade)
+            for r in to_grade:
+                by_type[r["item_type"].split(":")[0]] += 1
+
+    chunks = chunk(all_to_grade)
+    for i, ch in enumerate(chunks):
+        with (out / f"chunk_{i:03d}.csv").open("w", newline="") as f:
+            w = csv.DictWriter(f, fieldnames=AGENT_COLS, extrasaction="ignore")
+            w.writeheader()
+            w.writerows(ch)
+    (out / "manifest.json").write_text(
+        json.dumps(
+            {
+                "battery": want,
+                "n_chunks": len(chunks),
+                "n_rows": len(all_to_grade),
+                "by_item_type": dict(by_type),
+                "runs": runs_used,
+            },
+            indent=2,
+        )
+    )
+    print(f"{want}: {len(all_to_grade)} rows -> {len(chunks)} chunks  {dict(by_type)}")
+    print(f"  runs: {len(runs_used)}  out: {out}")
+
+
+if __name__ == "__main__":
+    main()

evals/judge.py

@@ -94,6 +94,37 @@ RUBRICS: dict[str, str] = {
         "The 'reference' shows the gold staff behavior. Pass if the BEHAVIOR "
         "matches the expected kind even if details differ."
     ),
+    "holistic": (
+        "ITEM TYPE = holistic tutor-answer quality (the staff-approval gate). "
+        "Decide as the course's expert tutor: would you approve sending this "
+        "answer to the student as-is? PASS only if ALL hold: (1) correct and not "
+        "misleading; (2) grounded in the right course/library material rather than "
+        "vague generic filler, for a course-content question; (3) it actually "
+        "helps the student LEARN -- explains or builds understanding at an "
+        "appropriate level, not just a bare dump, a wall of unexplained code, or "
+        "an unhelpful deflection; (4) appropriate scope and tone -- answers what "
+        "was asked, redirects platform/billing issues instead of inventing fixes, "
+        "acknowledges feedback without overpromising. FAIL if it is wrong, "
+        "ungrounded where grounding was needed, unhelpful for learning, or "
+        "off-scope. Judge the WHOLE answer, not a single claim; minor stylistic "
+        "nits alone are not a fail. Use confidence=low when it is a genuine "
+        "borderline 'I could see staff going either way'."
+    ),
+    "faithfulness": (
+        "ITEM TYPE = faithfulness / groundedness. The reference block is the "
+        "RETRIEVED EVIDENCE the tutor's own tools returned (corpus retrieval + KB "
+        "browse). Grade ONLY whether the answer's substantive factual and "
+        "technical claims are supported by that evidence. PASS if every "
+        "substantive claim is supported by, or a reasonable paraphrase/synthesis "
+        "of, the evidence. FAIL if the answer asserts specifics that are NOT in "
+        "the evidence -- fabricated API/function/parameter names, version numbers, "
+        "citations or URLs, course/lesson facts, or numeric claims (i.e. "
+        "hallucination beyond what the evidence supports). Widely-known general "
+        "programming facts an expert would accept need not appear in the evidence; "
+        "course-specific and library-specific specifics MUST. If the evidence is "
+        "empty or clearly insufficient to support the answer's specific claims, "
+        "FAIL (confidence=low if it is borderline)."
+    ),
     "replay_reply": (
         "ITEM TYPE = replay. The criterion asks whether this reply is about as "
         "helpful and correct as the real staff reply (shown as 'reference'). Pass "
@@ -127,9 +158,16 @@ def build_prompt(row: dict[str, str]) -> str:
         f"\nCRITERION:\n{row.get('criterion', '')}",
     ]
     if (row.get("reference") or "").strip():
-        parts.append(
-            f"\nSTAFF REFERENCE (context only, never shown to the tutor):\n{row['reference']}"
+        # For faithfulness the reference block IS the retrieved evidence to grade
+        # the answer's grounding against; for every other item type it is the
+        # staff answer, shown as context only.
+        label = (
+            "RETRIEVED EVIDENCE the tutor's tools returned (grade the answer's "
+            "grounding against THIS)"
+            if row["item_type"] == "faithfulness"
+            else "STAFF REFERENCE (context only, never shown to the tutor)"
         )
+        parts.append(f"\n{label}:\n{row['reference']}")
     parts.append(f"\nANSWER TO GRADE:\n{row.get('answer') or '(EMPTY)'}")
     return "\n".join(parts)
 

evals/report.py

@@ -97,6 +97,23 @@ def metric_rows(battery_type: str, runs: list[dict[str, Any]]) -> list[list[str]
     def add(name: str, fn) -> None:
         rows.append([name] + [fn(run["grades"]) for run in runs])
 
+    def src_tag(source_key: str) -> str:
+        """Per-metric provenance tag across the shown runs: [judge]/[human]/[mixed].
+
+        Lets a single report mix human-graded rows (e.g. Part B key-points,
+        session probes) with judge-graded rows (the new holistic/faithfulness)
+        and label each honestly, even when a run's overall grade_source is mixed.
+        """
+        seen = {
+            g.get(source_key)
+            for run in runs
+            for g in run["grades"]
+            if g.get(source_key)
+        }
+        if not seen:
+            return ""
+        return f" [{seen.pop()}]" if len(seen) == 1 else " [mixed]"
+
     add("cases run", lambda g: str(len({x["unit_id"] for x in g})))
     add("errors", lambda g: str(sum(1 for x in g if x.get("error"))))
     # --- Work & cost: run-time-INDEPENDENT, the unconfounded headline ----------
@@ -212,8 +229,24 @@ def metric_rows(battery_type: str, runs: list[dict[str, Any]]) -> list[list[str]
             "behavior proxy from code checks",
             lambda g: rate(col(g, "behavior_heuristic")),
         )
-        add("behavior (graded)", lambda g: rate(col(g, "behavior_pass")))
-        add("key-point coverage (graded)", lambda g: _kp_coverage(g))
+        add(
+            f"behavior (graded){src_tag('behavior_source')}",
+            lambda g: rate(col(g, "behavior_pass")),
+        )
+        add(
+            f"key-point coverage (graded){src_tag('key_points_source')}",
+            lambda g: _kp_coverage(g),
+        )
+        add(
+            f"holistic: staff-approval (graded){src_tag('holistic_source')}",
+            lambda g: rate(col(g, "holistic_pass")),
+        )
+        # NOTE: faithfulness-to-evidence is intentionally not reported. On bundles
+        # recorded under the old 6k tool-output cap, KB-browse evidence is
+        # truncated, so the judge can't see what the agent grounded on and the
+        # score tracks capture-completeness, not grounding (evals.md F23/F24
+        # class). grade.py only emits it once evidence is captured in full; re-add
+        # the row after raising TOOL_OUTPUT_MAX_CHARS and re-recording a run.
 
     if battery_type == "sessions":
         add("cumulative input tok (last turn, mean)", _final_cumulative)
@@ -224,9 +257,15 @@ def metric_rows(battery_type: str, runs: list[dict[str, Any]]) -> list[list[str]
             ),
         )
         add(
-            "probe accuracy (graded)",
+            f"probe accuracy (graded){src_tag('probe_source')}",
             lambda g: rate([x.get("probe_pass") for x in g if x.get("is_probe")]),
         )
+        # Answer quality across all session turns (not just probes): does holistic
+        # quality / faithfulness degrade under compaction over a long session?
+        add(
+            f"holistic: staff-approval (graded){src_tag('holistic_source')}",
+            lambda g: rate(col(g, "holistic_pass")),
+        )
         for probe_type in sorted(
             {
                 x.get("probe_type")
@@ -255,9 +294,16 @@ def metric_rows(battery_type: str, runs: list[dict[str, Any]]) -> list[list[str]
             "anti-pattern failures",
             lambda g: str(sum(1 for x in g if x.get("anti_pattern_hits"))),
         )
+        add(
+            f"holistic: staff-approval (graded){src_tag('holistic_source')}",
+            lambda g: rate(col(g, "holistic_pass")),
+        )
 
     if battery_type == "replay":
-        add("replay reply pass (human grade)", lambda g: rate(col(g, "replay_pass")))
+        add(
+            f"replay reply pass (graded){src_tag('replay_source')}",
+            lambda g: rate(col(g, "replay_pass")),
+        )
     return rows
 
 

evals/run_battery.py

@@ -38,9 +38,15 @@ from .common import append_jsonl, detect_battery_type, load_jsonl, write_jsonl
 
 logger = logging.getLogger("evals.run_battery")
 
-# Full tool outputs can be 40k chars each; keep bundles browsable. The full
-# size is preserved in output_chars so truncation is visible.
-TOOL_OUTPUT_MAX_CHARS = 6_000
+# Capture the full tool output the agent saw: the KB shell returns up to
+# DEFAULT_MAX_OUTPUT_CHARS = 40k chars/call (app/kb_shell.py), so 40k preserves
+# everything the model grounded on. This matters for offline faithfulness
+# grading -- under the old 6k cap, KB-browse evidence was truncated, so a
+# faithfulness judge scored capture-completeness, not grounding (evals.md
+# F23/F24 class). output_chars still records the true length, so any rare
+# overflow stays visible. Bundles are bigger (KB-heavy turns ~40k x several
+# calls), which is an acceptable cost for local-only runs.
+TOOL_OUTPUT_MAX_CHARS = 40_000
 # Slowest observed turn is ~2.5 min; anything past this is a wedged stream
 # (e.g. laptop sleep killed the connection mid-turn). The turn records a
 # TimeoutError and the unit re-runs on resume instead of hanging forever.

evals_slm_compaction.md → evals/slm_compaction.md

@@ -65,7 +65,7 @@ window is itself truncated at build time.
 
 **Naming caveat:** `full_context` here is *document-stuffing* (the whole lesson in
 the prompt), which is **not** `full_history`'s conversational-memory role. The
-sibling Gemini study (`evals_compaction.md`, F29) tests the same document-in-prompt
+sibling Gemini study (`evals/compaction.md`, F29) tests the same document-in-prompt
 idea but names that arm `full_history`, because it parks the lesson in turn 0 of a
 session — so don't read F29's `full_history` as the Axis-A memory policy.
 
@@ -137,7 +137,7 @@ Run on a **MacBook Pro, Apple M1 Pro, 16 GB**, one model at a time via
 3. **GraphRAG never wins outright — same verdict as the cloud experiment.** It ties
    `rag` on the qwens (100%) and edges it on llama (87% vs 80%), but at **~2.8× the
    context tokens (8.2k vs 2.9k) and ~2.5× the latency (~50-78s vs ~20-26s)**.
-   No quality gain justifies the cost (cf. `evals_graphrag.md`).
+   No quality gain justifies the cost (cf. `evals/graphrag.md`).
 
 4. **Summarization compaction degrades hard, worst on the smallest model.**
    `summary` and `hierarchical_summary` are the bottom methods (llama `summary`

pyproject.toml

@@ -31,7 +31,7 @@ dependencies = [
 # Eval-only: the GraphRAG-vs-RAG experiment (F28). Pulls graphrag + lancedb/litellm/
 # pandas, all imported lazily and only by the opt-in `graphrag` retriever — never on
 # the production path. Kept out of the default install so the Spaces image stays lean.
-# Reproduce the experiment with: uv sync --extra graphrag  (see evals_graphrag.md).
+# Reproduce the experiment with: uv sync --extra graphrag  (see evals/graphrag.md).
 graphrag = ["graphrag>=3.1.0"]
 
 [dependency-groups]

tests/test_build_public_docs_bundle.py

@@ -0,0 +1,168 @@
+from __future__ import annotations
+
+import json
+import pickle
+from pathlib import Path
+
+from app import kb_manifest
+from data.scraping_scripts import build_public_docs_bundle as builder
+
+# Use real registry keys so the COURSE/_DOC split matches production.
+COURSE_KEY = "master_ai_for_work"
+DOC_KEY = "transformers"
+
+
+def _write_jsonl(path: Path, rows: list[dict]) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("w", encoding="utf-8") as handle:
+        for row in rows:
+            handle.write(json.dumps(row) + "\n")
+
+
+def test_filter_jsonl_drops_course_rows(tmp_path: Path) -> None:
+    path = tmp_path / "headings.jsonl"
+    _write_jsonl(
+        path,
+        [
+            {"source": DOC_KEY, "heading": "Install"},
+            {"source": COURSE_KEY, "heading": "Lesson 1"},
+            {"source": DOC_KEY, "heading": "Usage"},
+        ],
+    )
+    kept = builder._filter_jsonl(path)
+    assert kept == 2
+    rows = [json.loads(line) for line in path.read_text().splitlines()]
+    assert {row["source"] for row in rows} == {DOC_KEY}
+
+
+def test_filter_tsv_drops_course_rows_keeps_header(tmp_path: Path) -> None:
+    path = tmp_path / "symbols.tsv"
+    path.write_text(
+        "symbol\tsource\ttitle\tpath\theading\tdoc_id\n"
+        f"AutoModel\t{DOC_KEY}\tT\tp\th\td1\n"
+        f"Lesson\t{COURSE_KEY}\tT\tp\th\td2\n",
+        encoding="utf-8",
+    )
+    kept = builder._filter_tsv(path)
+    assert kept == 1
+    lines = path.read_text().splitlines()
+    assert lines[0].startswith("symbol\tsource")  # header preserved
+    assert COURSE_KEY not in path.read_text()
+    assert DOC_KEY in path.read_text()
+
+
+def test_stage_kb_prunes_course_content(tmp_path: Path) -> None:
+    source_dir = tmp_path / "data"
+    kb = source_dir / "kb"
+    (kb / "raw" / "docs" / DOC_KEY).mkdir(parents=True)
+    (kb / "raw" / "docs" / DOC_KEY / "a.md").write_text("# A", encoding="utf-8")
+    (kb / "raw" / "courses" / COURSE_KEY).mkdir(parents=True)
+    (kb / "raw" / "courses" / COURSE_KEY / "l1.md").write_text("# L1", encoding="utf-8")
+    (kb / "wiki" / "courses").mkdir(parents=True)
+    (kb / "wiki" / "courses" / "c.md").write_text("course", encoding="utf-8")
+    (kb / "wiki").joinpath("index.md").write_text("# Index", encoding="utf-8")
+
+    generated = kb / "generated"
+    generated.mkdir(parents=True)
+    _write_jsonl(
+        generated / "corpus_manifest.jsonl",
+        [
+            {"source": DOC_KEY, "source_group": "docs", "path": "data/kb/raw/docs/x"},
+            {"source": COURSE_KEY, "source_group": "courses", "path": "y"},
+        ],
+    )
+    _write_jsonl(
+        generated / "headings.jsonl",
+        [{"source": DOC_KEY}, {"source": COURSE_KEY}],
+    )
+    (generated / "symbols.tsv").write_text(
+        "symbol\tsource\ttitle\tpath\theading\tdoc_id\n"
+        f"X\t{DOC_KEY}\tT\tp\th\td1\n"
+        f"Y\t{COURSE_KEY}\tT\tp\th\td2\n",
+        encoding="utf-8",
+    )
+
+    stage_dir = tmp_path / "stage"
+    stage_dir.mkdir()
+    summary = builder.stage_kb(source_dir, stage_dir)
+
+    staged_kb = stage_dir / "kb"
+    assert not (staged_kb / "raw" / "courses").exists()
+    assert (staged_kb / "raw" / "docs" / DOC_KEY / "a.md").exists()
+    assert not (staged_kb / "wiki" / "courses").exists()
+    assert (staged_kb / "wiki" / "index.md").exists()  # docs wiki kept
+
+    manifest = [
+        json.loads(line)
+        for line in (staged_kb / "generated" / "corpus_manifest.jsonl")
+        .read_text()
+        .splitlines()
+    ]
+    assert {row["source"] for row in manifest} == {DOC_KEY}
+    assert summary["manifest_rows"] == 1
+    assert COURSE_KEY not in (staged_kb / "generated" / "symbols.tsv").read_text()
+
+
+def test_rebuild_retrieval_pkls_drops_course_documents(tmp_path: Path) -> None:
+    source_dir = tmp_path / "data"
+    rows = [
+        {
+            "doc_id": "doc-transformers",
+            "name": "Transformers",
+            "url": "https://example.com/t",
+            "source": DOC_KEY,
+            "content": "# Heading\n\nUse AutoModel to load weights.",
+            "retrieve_doc": True,
+            "tokens": 9,
+        },
+        {
+            "doc_id": "doc-course",
+            "name": "Course Lesson",
+            "url": "https://example.com/c",
+            "source": COURSE_KEY,
+            "content": "# Lesson\n\nPrivate course material.",
+            "retrieve_doc": True,
+            "tokens": 7,
+        },
+    ]
+    _write_jsonl(source_dir / "all_sources_data.jsonl", rows)
+    (tmp_path / "stage" / builder.VECTOR_DB_DIR).mkdir(parents=True)
+
+    summary = builder.rebuild_retrieval_pkls(source_dir, tmp_path / "stage")
+    assert summary["documents"] == 1
+
+    dict_path = tmp_path / "stage" / builder.VECTOR_DB_DIR / builder.DOCUMENT_DICT_FILE
+    with dict_path.open("rb") as handle:
+        document_dict = pickle.load(handle)
+    assert set(document_dict) == {"doc-transformers"}
+
+
+def test_public_allow_patterns_toggles_contextual() -> None:
+    base = builder.public_allow_patterns(include_contextual=False)
+    assert base == ["chroma-db-all_sources/**", "kb/**"]
+    with_ctx = builder.public_allow_patterns(include_contextual=True)
+    assert "all_sources_contextual_nodes.pkl" in with_ctx
+
+
+def test_available_source_keys_excludes_absent_sources(tmp_path: Path) -> None:
+    kb_dir = tmp_path / "kb"
+    generated = kb_dir / "generated"
+    generated.mkdir(parents=True)
+    # A docs-only manifest (the public bundle): courses are simply not present.
+    _write_jsonl(
+        generated / "corpus_manifest.jsonl",
+        [
+            {"doc_id": "1", "source": DOC_KEY, "source_group": "docs"},
+            {"doc_id": "2", "source": "langchain", "source_group": "docs"},
+        ],
+    )
+    kb_manifest._MANIFEST_CACHE.pop(str(kb_dir), None)
+    keys = kb_manifest.available_source_keys(str(kb_dir))
+    assert keys == frozenset({DOC_KEY, "langchain"})
+    assert COURSE_KEY not in keys
+
+
+def test_available_source_keys_none_when_manifest_missing(tmp_path: Path) -> None:
+    missing = tmp_path / "no_kb"
+    kb_manifest._MANIFEST_CACHE.pop(str(missing), None)
+    assert kb_manifest.available_source_keys(str(missing)) is None

tests/test_config.py

@@ -77,6 +77,50 @@ def test_complete_bundle_skips_download_and_caches_readiness(tmp_path: Path) ->
             assert snapshot_download.call_count == 0
 
 
+def _repo_not_found_error():
+    try:
+        from huggingface_hub.errors import RepositoryNotFoundError
+    except ImportError:  # older huggingface_hub
+        from huggingface_hub.utils import RepositoryNotFoundError
+    return RepositoryNotFoundError
+
+
+def test_download_bundle_uses_public_when_no_token() -> None:
+    with patch("huggingface_hub.get_token", return_value=None):
+        with patch("app.config._snapshot_bundle") as snapshot:
+            config._download_bundle()
+    snapshot.assert_called_once()
+    assert snapshot.call_args.args[0] == config.PUBLIC_VECTOR_DB_REPO_ID
+
+
+def test_download_bundle_uses_private_when_token_present() -> None:
+    with patch("huggingface_hub.get_token", return_value="tok"):
+        with patch("app.config._snapshot_bundle") as snapshot:
+            config._download_bundle()
+    snapshot.assert_called_once()
+    assert snapshot.call_args.args[0] == config.VECTOR_DB_REPO_ID
+
+
+def test_download_bundle_falls_back_to_public_on_no_access() -> None:
+    import requests
+
+    repo_error = _repo_not_found_error()
+    response = requests.Response()
+    response.status_code = 401
+    calls: list[str] = []
+
+    def side_effect(repo_id: str, *, token) -> None:
+        calls.append(repo_id)
+        if repo_id == config.VECTOR_DB_REPO_ID:
+            raise repo_error("no access", response=response)
+
+    with patch("huggingface_hub.get_token", return_value="tok"):
+        with patch("app.config._snapshot_bundle", side_effect=side_effect):
+            config._download_bundle()
+
+    assert calls == [config.VECTOR_DB_REPO_ID, config.PUBLIC_VECTOR_DB_REPO_ID]
+
+
 def test_ensure_kb_agents_md_writes_once_and_atomically(tmp_path: Path) -> None:
     _write_bundle_files(tmp_path)
 

tests/test_evals_grade.py

@@ -1,13 +1,21 @@
 from __future__ import annotations
 
+import csv
+import tempfile
 import unittest
+from pathlib import Path
 
 from evals.common import detect_battery_type, normalize_url, percentile
 from evals.grade import (
     behavior_heuristic,
+    evidence_is_complete,
+    faithfulness_evidence,
     grade_persona_question,
+    merge_handgrades,
+    quality_sheet_rows,
     retrieval_metrics,
 )
+from evals.judge import RUBRICS, build_prompt, rubric_for
 
 
 class CommonTests(unittest.TestCase):
@@ -147,5 +155,202 @@ class PersonaGradingTests(unittest.TestCase):
         self.assertTrue(result["needs_judgment"])
 
 
+class FaithfulnessEvidenceTests(unittest.TestCase):
+    def test_collects_retrieval_and_kb_evidence(self) -> None:
+        bundle = {
+            "tool_calls": [
+                {
+                    "tool_name": "retrieve_tutor_context",
+                    "args_text": "context engineering",
+                    "output_text": "Lesson 3 covers context engineering.",
+                    "matches": [{"title": "Lesson 3", "url": "https://x/l3"}],
+                },
+                {
+                    "tool_name": "run_kb_command",
+                    "args_text": "rg foo",
+                    "output_text": "raw/courses/agent/lesson.md: foo bar",
+                },
+            ]
+        }
+        evidence = faithfulness_evidence(bundle)
+        self.assertIn("context engineering", evidence)
+        self.assertIn("SOURCES: Lesson 3 <https://x/l3>", evidence)
+        self.assertIn("raw/courses/agent/lesson.md", evidence)
+
+    def test_empty_when_no_retrieval_tool(self) -> None:
+        bundle = {"tool_calls": [{"tool_name": "some_other_tool", "output_text": "x"}]}
+        self.assertEqual(faithfulness_evidence(bundle), "")
+        self.assertEqual(faithfulness_evidence({}), "")
+
+    def test_respects_max_chars(self) -> None:
+        bundle = {
+            "tool_calls": [{"tool_name": "run_kb_command", "output_text": "z" * 5000}]
+        }
+        self.assertEqual(len(faithfulness_evidence(bundle, max_chars=100)), 100)
+
+
+class EvidenceCompletenessTests(unittest.TestCase):
+    def test_complete_when_no_truncation(self) -> None:
+        bundle = {
+            "tool_calls": [
+                {"tool_name": "run_kb_command", "output_text": "abc", "output_chars": 3}
+            ]
+        }
+        self.assertTrue(evidence_is_complete(bundle))
+
+    def test_incomplete_when_kb_output_truncated(self) -> None:
+        # output_chars (true length) exceeds captured output_text => truncated.
+        bundle = {
+            "tool_calls": [
+                {
+                    "tool_name": "run_kb_command",
+                    "output_text": "x" * 6000,
+                    "output_chars": 40000,
+                }
+            ]
+        }
+        self.assertFalse(evidence_is_complete(bundle))
+
+    def test_non_retrieval_truncation_is_ignored(self) -> None:
+        bundle = {
+            "tool_calls": [
+                {"tool_name": "some_tool", "output_text": "x", "output_chars": 99999}
+            ]
+        }
+        self.assertTrue(evidence_is_complete(bundle))
+
+    def test_faithfulness_row_gated_on_complete_evidence(self) -> None:
+        truncated = {
+            "run_id": "r",
+            "battery_type": "singleturn",
+            "preset": "prod",
+            "query": "q",
+            "answer": "a",
+            "tool_calls": [
+                {
+                    "tool_name": "run_kb_command",
+                    "args_text": "rg foo",
+                    "output_text": "y" * 6000,
+                    "output_chars": 40000,
+                }
+            ],
+        }
+        types = {r["item_type"] for r in quality_sheet_rows(truncated)}
+        self.assertIn("holistic", types)
+        self.assertNotIn("faithfulness", types)  # truncated -> no faithfulness
+
+        full = dict(truncated)
+        full["tool_calls"] = [
+            {
+                "tool_name": "run_kb_command",
+                "args_text": "rg foo",
+                "output_text": "y" * 100,
+                "output_chars": 100,
+            }
+        ]
+        types_full = {r["item_type"] for r in quality_sheet_rows(full)}
+        self.assertIn("faithfulness", types_full)  # full evidence -> emitted
+
+
+class JudgeRubricTests(unittest.TestCase):
+    def test_holistic_and_faithfulness_rubrics_exist(self) -> None:
+        self.assertIs(rubric_for("holistic"), RUBRICS["holistic"])
+        self.assertIs(rubric_for("faithfulness"), RUBRICS["faithfulness"])
+
+    def test_unknown_item_type_falls_back_to_key_point(self) -> None:
+        self.assertIs(rubric_for("mystery"), RUBRICS["key_point"])
+
+    def test_faithfulness_reference_labeled_as_evidence(self) -> None:
+        row = {
+            "item_type": "faithfulness",
+            "question": "q",
+            "criterion": "grounded?",
+            "reference": "RETRIEVED_TEXT_X",
+            "answer": "a",
+        }
+        prompt = build_prompt(row)
+        self.assertIn("RETRIEVED EVIDENCE", prompt)
+        self.assertNotIn("STAFF REFERENCE", prompt)
+
+    def test_other_item_types_keep_staff_reference_label(self) -> None:
+        row = {
+            "item_type": "key_point",
+            "question": "q",
+            "criterion": "claim",
+            "reference": "STAFF_TEXT",
+            "answer": "a",
+        }
+        self.assertIn("STAFF REFERENCE", build_prompt(row))
+
+
+class MergeHandgradesTests(unittest.TestCase):
+    FIELDS = [
+        "sheet_row_id",
+        "run_id",
+        "item_type",
+        "criterion",
+        "grade",
+        "note",
+    ]
+
+    def _merge(self, grades, filled_rows):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "filled.csv"
+            with open(path, "w", encoding="utf-8", newline="") as f:
+                writer = csv.DictWriter(f, fieldnames=self.FIELDS)
+                writer.writeheader()
+                writer.writerows(filled_rows)
+            return merge_handgrades(grades, path)
+
+    def test_new_item_types_and_per_metric_source(self) -> None:
+        grades = [{"run_id": "r1"}]
+        filled = [
+            # human-graded key points
+            {
+                "sheet_row_id": "r1|key_point|a",
+                "run_id": "r1",
+                "item_type": "key_point",
+                "criterion": "k1",
+                "grade": "pass",
+                "note": "",
+            },
+            {
+                "sheet_row_id": "r1|key_point|b",
+                "run_id": "r1",
+                "item_type": "key_point",
+                "criterion": "k2",
+                "grade": "fail",
+                "note": "",
+            },
+            # judge-graded new dimensions
+            {
+                "sheet_row_id": "r1|holistic|c",
+                "run_id": "r1",
+                "item_type": "holistic",
+                "criterion": "h",
+                "grade": "pass",
+                "note": "[judge:high] good answer",
+            },
+            {
+                "sheet_row_id": "r1|faithfulness|d",
+                "run_id": "r1",
+                "item_type": "faithfulness",
+                "criterion": "f",
+                "grade": "fail",
+                "note": "[judge:low] fabricated param",
+            },
+        ]
+        merged = self._merge(grades, filled)[0]
+        self.assertEqual(merged["key_points_passed"], 1)
+        self.assertEqual(merged["key_points_total"], 2)
+        self.assertEqual(merged["key_points_source"], "human")
+        self.assertTrue(merged["holistic_pass"])
+        self.assertEqual(merged["holistic_source"], "judge")
+        self.assertFalse(merged["faithfulness_pass"])
+        self.assertEqual(merged["faithfulness_source"], "judge")
+        # human key-points + judge new dims => overall mixed
+        self.assertEqual(merged["grade_source"], "mixed")
+
+
 if __name__ == "__main__":
     unittest.main()