evals.md

# Evaluating the AI tutor

This document explains how we test the AI tutor with repeatable conversations and measured results. The main question is: when the tutor has a long chat history, which memory strategy gives the best answers without wasting too many tokens, dollars, or seconds?

The eval setup compares answer quality, memory retention, retrieval accuracy, tokens, cost, and latency for the June 2026 workshop. The same harness also becomes the ongoing quality program afterwards. Research sources and design rationale live in `evals/background.md`.

**Running a new experiment?** See `evals/contributing.md` for the step-by-step (run → upload data to HF → record an `F<N>` finding here → merge) and the definition-of-done checklist, so the next person inherits both your result and the data to reproduce it.

Plain-English map:
- A **battery** is a dataset of test conversations or test questions.
- A **run** is one battery executed with one model and one memory setting.
- A **trace bundle** is the saved JSON record for one tutor turn: the question, answer, tool calls, sources, token usage, timing, and any error.
- A **grade** is the pass/fail or metric row computed later from saved trace bundles.
- A **memory preset** is a named setting that controls how much chat history the tutor keeps, summarizes, or stores as a student profile.
- An **axis** is one independent dimension the experiment varies. Part B varied a single axis (the memory preset). Part C uses two: **A — memory & context management** (how conversation history is kept or compacted) and **B — retrieval & tool outputs** (how retrieved docs and tool results are sized, cleared, or whether the KB tool exists). A **single-axis** run changes one variant on one axis and holds everything else at the production baseline, so any difference is attributable to that one change; a **cross-product** tests every Axis-A × Axis-B combination at once (far more runs, far bigger bill).

## What we evaluate, and what varies

The system under test is the production tutor (`app/`): the same agent, retrieval tools, prompts, and telemetry used by the app. The tutor can answer from the course/docs corpus through `retrieve_tutor_context`, browse the local knowledge base through `run_kb_command`, and keep per-thread conversation memory.

**The single variable is how the tutor manages context** — 13 arms across two axes in the Part B/C screen (**A** = memory/history, **B** = retrieval/tool-outputs), with later experiments adding more arms (F29's summarization family, F35/F37's `exp_*` factorial + structured-prefix arms; `exp_c400/c800` built, unrun). Within each experiment everything else is held constant — system prompt, retrieval configuration, selected sources, web tools off, temperature, and the model. **The Part B/C screen held the model at `gemini-3.5-flash`; later experiments varied it** (DeepSeek-V4-Flash — now the production default — in F25/F26/F35–F38, Gemini 2.5 Flash in F29, three Ollama SLMs in F30/F31), with per-finding comparability caveats: absolute numbers never cross fleets. Most arms are a `MemoryConfig` preset (`app/memory_presets.py`, part of the agent cache key); a few vary a retrieval knob or a tool toggle instead. In this document, **compaction** means reducing the prompt by summarizing older messages or clearing old tool outputs.

**The arms — every strategy we tested:**

| Arm | Axis | Technique | What it tests | Phase |
|---|---|---|---|---|
| `full_history` | — | keep everything (no compaction) | quality/memory upper bound; token worst case | B + C anchor |
| `prod` | A+B | summarization + tool-result clearing | the live production baseline | B + C anchor, D |
| `aggressive` | A+B | early summarization + clearing | high-pressure compaction | B |
| `profile_memory` | A (store) | semantic profile store + prod compaction | long-term personalization | B, D rerun |
| `summarization_only` / `editing_only` | A / B | one compaction technique alone | isolate each half of prod's bundle | defined, folded/dropped (F3) |
| `sliding_window` | A | trimming / keep last N | recency-only memory | C |
| `prompt_compression` | A | rewrite history to fewer tokens | shrink text, drop no facts | C |
| `selective_retention` | A | constraint-preserving summarization | quality-preserving summary | C |
| `context_reset` | A | summary-seeded fresh state | aggressive prefix rewrite | C |
| `incontext_history_retrieval` | A | retrieve relevant old turns | retrieve vs carry/summarize all history | C |
| `clear_retrieval_kb` | B | clear tool outputs incl. retrieval/KB | the F3 fix (prod exempts them) | C |
| `observation_truncation` | B | head/tail-truncate tool outputs | trim the dominant token source | C |
| `retrieval_budget_30k` | B | retrieval token-budget knob (100k→30k) | does a tighter budget drop recall (F1) | C |
| `kb_off` | B | disable the KB browse tool | agentic browse vs top-k RAG | C |

**The datasets — what we ran the arms on** (full schemas in `data/eval/README.md`; detail in "The dataset" below):

| Battery | Size | Tests | What actually ran |
|---|---|---|---|
| `battery_singleturn_v1` | 60 one-shot Qs | answer quality + response type | B (all 60), C (24-case subset) |
| `battery_sessions_v1` | 32 sessions (113 probes) | memory under compaction | **only 3 sessions ever** (s01/s02/s08), in B and C |
| `battery_personas_v1` | 10 personas × 4 Qs | long-term profile memory | B (all) |
| `replay_n1_v1` | 30 "next staff reply" cases | multi-turn behavior on real threads | **built, never run** |
| `battery_sessions_v2` | 6 sessions, 3 tiers (contradiction / long-horizon / entity) | where `full_history` might lose | Gemini partial (`prod` all 6, `full_history` 2 Tier-1), then in full on DeepSeek: the 12-arm v2 matrix (F25/F26) and the ×3-trial stage-1 factorial (F35–F37). **Deprecated 2026-07-15** (filler confound, see harness corrections) |
| `battery_sessions_v2_1` | v2 with 7 recycled filler turns repaired (probes/plants identical) | same tiers, confound-free | built 2026-07-15; the required v2 successor for new runs |

**What actually ran, by phase:** **Part B** — the 4-arm bake-off (`full_history`, `prod`, `profile_memory`, `aggressive`) over the full single-turn + personas batteries + 3 sessions, ×2 trials (→ F1–F13). **Part C** — the 11-arm screen (the 2 anchors + 9 variants) over a 24-case single-turn subset + 3 sessions, ×1 trial (→ F14–F20, F23). **D** — re-ran `profile_memory` with its store actually engaged (→ F22). **v2** — a partial long-context / contradiction run (→ the open v2 items in "Part C — what remains" below). Later phases moved off Gemini: the **DeepSeek v2 matrix** (F25/F26), **GraphRAG head-to-head** (F28), the **Gemini-2.5 compaction study** (F29), the **SLM studies** (F30/F31), and the **DeepSeek stage-1 factorial + structured-compaction follow-up** (F35–F38, ×3 trials on the v2 sessions battery). Nothing has been run at promotion grade (full batteries × 3 trials); the earlier phases are 1–2-trial screens.

*Why the B→C order:* Part B's findings drove it — tokens live in tool outputs not history (F1/F3) and compaction can cost more than it saves (F2/F9), so the cheap retrieval/tool-output variants ran first and the history-compaction variants ran to confirm that inversion. That narrative lives in the Results sections below; this section is the complete catalog.

## What we measure

| Layer | Metrics | Source |
|---|---|---|
| Runtime and cost | Time to first answer text, total turn time, input/output tokens, cached tokens, estimated dollars, number of model calls, and whether compaction fired. | The `context_stats` event emitted at the end of each turn (`app/telemetry.py`). This does not depend on LangSmith. |
| Tool behavior | Which tools the tutor called, how often, and whether it re-searched for information it had already seen. | Tool-call events saved in each trace bundle. |
| Retrieval | Whether the retrieved chunks included the correct course or lesson. `recall@shown` means "was the right source among the results shown to the model?" `MRR` rewards putting the right lesson earlier in the shown results. | Ground truth stored on each real discussion case: `source_key` and `lesson_url`. |
| Response behavior | Whether the tutor did the correct kind of thing: answer from course content, answer generally, redirect a platform/support issue, or acknowledge feedback. | Cheap code checks for early signals; final behavior accuracy comes from human grades. |
| Answer quality | Whether the answer covered the required key points, passed graded memory probes, and (holistic) whether course staff would approve sending it as a genuinely helpful, learning-oriented reply. A separate **faithfulness** check (every claim grounded in retrieved evidence) is built but parked — see the cap correction below. | Human pass/fail grading in a blinded workbook, or the validated blind subagent judge (same per-item rubric). LLM judges are used only after they are validated against human labels. Holistic and probe quality are complementary — holistic is graded blind to conversation history, so it cannot see a memory failure (F34); read them together. |
| Memory | Whether the tutor remembered or updated facts from earlier in the session, followed preferences, resolved references like "that thing from earlier," and used stored student profiles. | Session probes and persona checks. |

Cost is reported both as raw tokens and as estimated dollars after cached-token discounts. These can rank presets differently (finding F2).

## The dataset

Source: `data/academy_discussion_eval.jsonl` — 151 real student posts from the academy discussion boards with real staff answers, annotated by gemini-3.5-flash. All eval inputs are real or hand-authored; **we never generate reference answers** — ground truth is staff replies (distilled to `key_points`) or facts we wrote ourselves.

**Review.** All 62 gold + 30 not-time-bound usable annotations (92) were re-reviewed against the live KB with file/line evidence. 60 kept, 32 excluded. Main corrections: the annotator badly under-flagged staleness (course content was updated after many posts — ~14 exclusions had premises no longer true in today's corpus), 5 near-duplicates, 1 fabricated URL, 1 misaligned question/answer pair, and key points rewritten to claims that are atomic, still true, and binary-checkable. Colab-notebook dependency was checked per case: **1 of 92** truly required the notebook (excluded); lessons embed the cells everywhere else. Full audit trail: `data/eval/review_log_v1.md` (11 judgment-call cases flagged for human review).

**Batteries** (`data/eval/`, schemas and glossary in its `README.md`; files are gitignored because they contain real student text):

| Battery | Contents | Tests |
|---|---|---|
| `battery_singleturn_v1` | 60 reviewed real questions asked as one-off chats. They include course-content questions, support/platform issues, general AI/programming questions, and course feedback. | Basic answer quality and correct response type. Memory presets should score about the same here; big differences suggest a bug. |
| `battery_sessions_v1` | 32 multi-turn study sessions. Early turns give the tutor facts about the student, middle turns make the chat long, and later turns check whether important facts survived. | Memory under pressure. A **probe** is one of the later turns that gets graded. Probe examples: remember a fact, respect a preference, understand "that thing from earlier," or use a newer fact after the student changed their mind. |
| `battery_personas_v1` | 10 stored student profiles with 4 questions each. The questions are written so the best answer needs the profile. | Long-term profile memory: does a stored profile actually improve personalization across fresh chats? |
| `replay_n1_v1` | 30 tests made from real multi-turn discussion threads. For each test, we give the tutor the conversation up to just before a real staff reply, ask it to write the next reply, and compare it with what staff actually wrote. | Multi-turn behavior on fully real data. This is secondary because it needs human or validated judge grading. |

## How it runs

```bash
uv run -m evals.run_battery --battery data/eval/battery_singleturn_v1.jsonl --preset prod --trials 2 --out runs/exp
uv run -m evals.grade  --run runs/exp                      # code checks + handgrade_sheet.csv
uv run -m evals.check_triggers --runs runs/exp             # gate: compaction fired where probes assume
uv run -m evals.report --runs runs/expA runs/expB          # side-by-side tables + token curves
uv run -m evals.handgrade_workbook build|merge ...         # blinded human-grading workbook
```

Command roles:
- `run_battery` talks to the tutor and saves one JSON trace bundle per turn.
- `grade` reads saved bundles and writes automatic grades plus a CSV for human judgments.
- `check_triggers` verifies that session probes really happened after compaction. A memory test is not useful if memory trimming never occurred.
- `report` builds side-by-side tables and token curves from graded runs.
- `handgrade_workbook` builds and merges a blinded grading workbook, so the grader does not know which preset produced an answer.

Every turn persists a JSON trace bundle, so grading and reporting can re-run offline without touching the API. Runs are resume-safe: re-run the same command after an interruption and completed units are skipped. Each turn has a 10-minute timeout, and LangSmith is off by default. Running the tutor needs `COHERE_API_KEY` plus the key for the model under test — `DEEPSEEK_API_KEY` for today's default (with only a Gemini key the app substitutes gemini-2.5-flash outright and the harness flags every bundle); the Part B/C runs used `GEMINI_API_KEY`; the full 4-preset comparison run below cost **≈$323** at corrected pricing (the pre-2026-06-13 price table under-priced `gemini-3.5-flash` ~4.4×; the often-quoted "$73" is a pre-correction figure — see Harness corrections).

**Setup for collaborators.** Code and docs are in git; the datasets and run results contain real student text and live only in the private HF dataset (`towardsai-tutors/ai-tutor-data` — git is force-pushed to the public prod Space on deploys, so student data never enters git). With an `HF_TOKEN` that can read it:

```bash
export HF_TOKEN=hf_...    # token with READ access; the inline cmd does NOT load .env (or: uv run huggingface-cli login)
mkdir -p data/eval runs/_grading  # runs/ is gitignored, so a fresh clone has no runs/ dir yet
uv run python -c "from huggingface_hub import snapshot_download as d; d(repo_id='towardsai-tutors/ai-tutor-data', repo_type='dataset', allow_patterns=['eval/**','eval_runs/**'], ignore_patterns=['eval/README.md'], local_dir='.')"
mv eval/* data/eval/ && mv eval_runs/part_*/* runs/   # restore working paths (part_b, part_c, ...)
mv eval_runs/slm_compaction/axis_a/* runs/ 2>/dev/null   # SLM Axis-A runs -> runs/axisa_* (F31); Axis-B (F30) stays under eval_runs/slm_compaction/axis_b/, see evals/slm_compaction.md
mv eval_runs/deepseek_compaction_stage1/deepseek_* runs/ 2>/dev/null && mv eval_runs/deepseek_compaction_stage1/_grading/* runs/_grading/ 2>/dev/null   # DeepSeek stage-1 compaction runs + judge verdicts (F35-F38)
```

## Results — Part B comparison run, 2026-06-12

4 presets × 2 trials over the full single-turn battery, full persona battery, and 3 sessions (s01 13-turn, s02 agentic, s08 fact-update). 1,232 turns, zero API errors. Tables: `runs/b_report/report.md`; token curves: `tokens_by_turn.csv`. **Auto-graded only** means the current table includes metrics computed by code. Answer quality and session probe accuracy appear after the human grading workbook is filled and merged.

| | full_history | prod | profile_memory | aggressive |
|---|---|---|---|---|
| personalization pass (personas, n=70) | 63% | 67% | **94%** | 56% |
| sessions: est. cost/turn (corrected pricing) | **$0.112** | $0.237 | $0.218 | $0.296 |
| sessions: median time to first answer text | **17s** | 21s | 22s | 43s |
| sessions: tool calls/turn | **2.8** | 3.5 | 3.9 | 8.3 |
| sessions: cumulative input tokens | 2.9M | 1.7M | 1.6M | 1.75M |
| single-turn: behavior proxy from code checks (n=106) | **88%** | 86% | 86% | 78% |
| single-turn: retrieval recall@shown source | 51% | 50% | 49% | 47% |

Row definitions:
- **personalization pass** — % of persona question-runs whose answer passed every authored check (expected regex matched, e.g. `conda` for the conda persona) with no anti-pattern hit (e.g. bash `export` for a Windows persona). n=70: 40 questions × 2 trials, minus 10 runs whose checks need human judgment.
- **est. cost/turn** — mean estimated $ per conversation turn: token counts × `MODEL_PRICING`, cached input billed at the cache-read discount (our table, not the invoice).
- **median time to first answer text** — median seconds from user message to first visible answer text. It includes tool calls and internal model rounds before the visible answer starts, so it approximates the user's perceived wait.
- **tool calls/turn** — mean retrieval + KB-command invocations per turn; here it's the re-work signal (compaction presets re-search for evidence their compressed history lost).
- **cumulative input tokens** — all input tokens billed across an entire session (every turn, every internal call), mean over the 6 session-runs per preset.
- **behavior proxy from code checks** — % of single-turn case-runs where a cheap code check confirms the right *kind* of response (course question → called retrieval/KB; support issue → points to support; feedback → acknowledges). n=106 of 120: `answer_general` has no proxy check. This is only an early signal; authoritative behavior accuracy comes from human grades.
- **retrieval recall@shown source** — % of case-runs where the reranked retrieval results the agent saw contained at least one chunk from the correct course. Stricter `recall@lesson` (right lesson page, 30–36%) and the right-lesson ranking score (MRR) are in the full report.

### Observations

Numbered findings; each states what it was tested on. Convention: entries are never edited, only superseded.

- **F1 — Retrieval payloads dominate input tokens, not conversation history.** Each retrieval call may return up to `DEFAULT_CONTEXT_TOKEN_BUDGET = 100_000` tokens; turns average ~200k input. (All runs; high confidence.) → retrieval budget became a Part C variant dimension.
- **F2 — Compaction saves tokens but not necessarily dollars.** Summarization rewrites the prompt prefix and invalidates Gemini's implicit cache: on the same session, `full_history` had 86.8% of input billed at the ~4x cache discount; `aggressive` used 44% fewer tokens yet cost 68% more. (s03 × 3 presets, then confirmed n=24 session-runs; provider-specific — Anthropic caching is explicit.)
- **F3 — Clearing old tool-output messages never fires on this workload.** `ClearToolUsesEdit` excludes retrieval results — where the tokens are; 0 clears across all session runs. → `editing_only` dropped from Part B; "clear retrieval results too" variant queued for Part C.
- **F4 — Aggressive compaction degrades even single-turn behavior.** 18.0 vs 9.6 LLM calls/turn, 57s vs 39s median, behavior proxy −10pts vs full_history (60 cases × 2 trials). Mid-turn compaction churn changes agent behavior, it doesn't just trim.
- **F5 — Gemini reasoning tokens are ~90%+ of output even with reasoning display off.** Billed as output; dominates latency; recorded per model in `usage_by_model`.
- **F6 — Long context costs latency, not correctness.** Zero API errors in 1,018+ turns at any size (max 6.06M tokens across one turn's calls; largest single context 274k). Median time to first answer text scales 22s → 76s from <100k to >800k input tokens/turn. Compaction's value here is responsiveness and spend, not keeping the model functional.
- **F7 — ~0.8% of turns produce no answer text** despite tool calls and billed reasoning tokens; preset- and size-independent. Candidate golden-case assertion ("answer non-empty").
- **F8 — Profile memory wins on quality AND cost.** 94% personalization vs 56–67% without, while the cheapest persona preset ($0.049 vs $0.081–0.095/turn, 7.6 vs 10–11.2 tool calls): the stored profile saves the agent from re-searching for user context. (40 questions × 2 trials × 4 presets, auto checks; LLM-check rows pending human grades.)
- **F9 — Full history is cheapest AND fastest up to 13 turns; compaction causes re-work.** Sessions: $0.034/turn, time to first answer text 17s, 2.8 tool calls for `full_history` vs $0.051–0.066, 21–43s, 3.5–8.3 elsewhere — despite ~2x the tokens (F2's cache mechanism plus a second one: raw history lets the agent re-use earlier retrieval evidence; summaries force re-retrieval). (3 sessions × 4 presets × 2 trials, n=24, zero errors.) The conventional pitch inverts: under modern prompt caching, the naive baseline wins short-to-medium sessions; compaction must justify itself on quality (pending human grades) and long horizons. Where the crossover actually sits is a Part C question.
- **F10 — Compaction degrades within-session memory, and what it drops is *old* facts.** Session-probe accuracy: `full_history` **92%** vs `prod` 38% / `profile_memory` 38% / `aggressive` 42% (n=24/preset). The collapse is entirely turn-0 material — `fact_recall` 100% → 17–25%, `preference_compliance` 100% → 0% — while `fact_update` stays **100% across all presets** (the mid-session update sits inside the kept-recent window; summarization evicts the early planted facts, not the recent ones). With F9 (full_history also cheapest and fastest here), the naive baseline wins cost, speed, AND memory on ≤13-turn sessions — the workshop headline. (3 sessions × 4 presets × 2 trials; **provisional — LLM-graded under 4 reviewer rubric policies, pending human review of the 96 probes**.)
- **F11 — Profile memory helps personalization, not working memory.** `profile_memory` reached 92% persona personalization (vs 54–66% without) yet scored **38% on session probes — identical to `prod`**: the long-term student-profile store improves fresh-chat answers while the live thread is still summarized exactly as in `prod`. Long-term store ≠ in-session retention; they are independent subsystems and a preset can win one while tying the other. (Personas n=80 auto; sessions n=24, provisional.) Refines F8.
- **F12 — Aggressive compaction is dominated on every axis.** Beyond F4's cost/latency blowup (18 LLM calls/turn, 57s median), it is also worst on quality: key-point coverage 36% vs 72% (`full_history`), single-turn behavior 67% vs 83–92%, session probes 42% vs 92%. No measured metric favors it. (Single-turn quality n thin at 12–25; sessions n=24; provisional.)
- **F13 — Session-probe grades are human-confirmed (zero overrides), so F10–F12's memory numbers are no longer provisional.** 2026-06-15 Omar reviewed all 96 session probes — the 83 high-confidence LLM verdicts by sign-off and the 13 low-confidence ones individually — and agreed with every grade. The provisional LLM grading therefore stands as human ground truth, confirming the 92%-vs-38% probe-accuracy result (F10) and the probe components of F11/F12. Caveat: only the session-probe tier was human-reviewed; the non-probe quality rows (persona, key_point, behavior — feeding key-point coverage and single-turn behavior in F8/F12) remain LLM-graded. The judge has **not** yet been validated against these labels (next step), so judge-graded Part C numbers stay gated.

### Part C screen (2026-06-15)

11 arms (prod + full_history anchors + 9 variants), each on a fixed subset (24 stratified single-turn + 3 sessions s01/s02/s08), 1 trial, ~660 turns, 0 errors, ≈$163 at correct pricing. Graded by the validated subagent judge. Full table: `runs/c_report/report.md`. Probe-accuracy n=12/arm (1 trial) — treat percentages as coarse rankings, not precise rates; promotion (×3 trials, full batteries) would firm them up.

- **F14 — The LLM judge is validated; Part C quality columns are reportable.** A blind subagent re-grade of the 96 human-confirmed probes reproduced them at 98% agreement / TPR 100% / TNR 96% (clears the >90% gate; this measures grader *reproducibility*, since the labels were sign-offs, not blind-from-scratch). The validated grader is the subagent workflow (same per-item-type rubric as `evals/judge.py`), run on subscription at no API cost; all Part C arms are graded that way. Supersedes F13's "judge not yet validated" caveat. (`runs/b_report/judge_val/`.)
- **F15 — The screen reproduces F9/F10 on independent arms.** `full_history` is cheapest on sessions ($0.10/turn) AND best memory (100% probe); every compaction arm is both pricier and weaker. `prompt_compression` also reaches 100% memory — it shrinks message text but drops no facts — yet saves little (highest tokens), so "don't drop content" preserves memory but is not a cost win. (Screen; n thin.)
- **F16 — `incontext_history_retrieval` works but is short-session-neutral.** 83% probe accuracy (best after full_history): retrieving the relevant old turns restores what summarization loses. But on ≤13-turn sessions it can't drop much, so it costs ≈ full_history (~$0.20/turn) plus embed overhead; its cost payoff needs LONG sessions (a `_v2` battery). The principled cost answer to F9, pending a long-horizon test; separate from the higher-priority v2 contradiction-precision question.
- **F17 — Turning the KB off inverts retrieval recall (Axis B tradeoff).** `kb_off` forces `retrieve_tutor_context` instead of browsing → single-turn recall@shown jumps 50%→96%, but it is the priciest session arm ($0.31/turn) and worst memory (33%). KB browsing is cheaper and better for memory yet actively *lowers* top-k retrieval recall (the agent browses instead of retrieving the labeled source). Not a clear win either way.
- **F18 — The 100k retrieval budget is over-provisioned.** `retrieval_budget_30k` matches prod's recall@shown (50%) at a third of the budget → tokens can be cut with no recall loss on this subset (direct confirmation of F1). The 10k rung is untested.
- **F19 — `observation_truncation` backfires (Axis B negative).** Head/tail-truncating tool outputs in the model's view makes the agent re-call tools to recover what was cut: 22.8 LLM calls/turn, 430s p95 latency (single-turn), memory only 33% — the same churn pathology as `aggressive` (F4).
- **F20 — Axis-A summary/trim arms do not rescue memory.** `context_reset` (17% probe, 56% key-points — dominated on every axis), `selective_retention` (25%), and `sliding_window` (42%) all stay well below `full_history`/`prompt_compression` and none beats `prod` (58%): dropping or summarizing early turns loses the planted facts the probes test (consistent with F10).

- **F21 (2026-06-16) — Supersedes F11 in part: `profile_memory` was dormant on the sessions battery, so its 38% is `prod`, not a test of the profile.** `evals/run_battery.run_session` passes no `student_id`, and `StudentProfileMiddleware` injection plus the post-turn write-back both no-op without one — so on sessions `profile_memory` reduces exactly to `prod` and could not differ. The 38% is real but comes from the live conversation thread + prod's lossy summary (the student states the facts in-thread; summarization keeps the recent ones and a compressed summary of the rest), **not** from any profile — which is why it is ~38%, not 0. F11's reading that this shows "long-term store ≠ in-session retention, independent subsystems" is therefore unsupported: the store was never engaged in-session. Whether a store that captures in-session facts verbatim and re-injects them recovers recall under compaction is **open** — the `collection_memory` v1 test (sessions get a `student_id` + an append-atomic-facts write-back; goalposts: compaction 17–25% vs `full_history` 100%). F11's personas number (94%) stands; personas do set a `student_id`.

- **F22 (2026-06-16) — Activating `profile_memory` on sessions rescues much of the old-fact loss, but does not replace full history.** Experiment A reran the three Part-C sessions with `run_session` passing a per-session/per-trial `student_id`, so the profile store was actually read/written and injected through the system prompt on every turn. With compaction active at 100% of probes, probe accuracy rose to **75% (9/12)** vs same-screen `prod` **58% (7/12)**, while `full_history` stayed **100% (12/12)**. The gain was concentrated exactly where F10/F21 predicted: `fact_recall` improved to **83% (5/6)** vs `prod` **33% (2/6)**, and `preference_compliance` to **100% (1/1)** vs `prod` 0%. It still missed two reference/consistency probes, and was not a cost win on this short screen ($0.265/turn vs `prod` $0.229 and `full_history` $0.103). Interpretation: injecting stored facts outside summarized history can recover many facts lost by compaction, but the current "5 durable lines" profile write-back is an incomplete working-memory store. Next test: `collection_memory` / verbatim atomic facts to determine whether the remaining gap is extraction/storage loss vs answer synthesis. **Caveat (thin n):** the only robust signal is `fact_recall` (5/6 vs 2/6) and the 9/12-vs-7/12 aggregate — the `preference_compliance`/`fact_update` cells are n=1 anecdotes; and `profile_memory` actually *regressed* on anaphora / anaphora_consistency (50% vs prod's 100%, n=2 each), possibly profile injection distracting from reference resolution, possibly noise. (Codex handgrade at Omar's request; one embeddings row was borderline-pass, strict fail would make this **67% (8/12)** without changing the direction.)

- **F23 (2026-06-17) — F17's recall inversion was a measurement artifact: `recall@shown` is blind to KB grounding, and a KB-fair metric erases the gap.** `recall@shown`/`recall@lesson`/MRR count only `retrieve_tutor_context` matches, so when `prod` grounds by *browsing* the KB instead of retrieving, the labeled source is invisible to the metric — which is exactly `kb_off`'s only structural difference, so the 50%→96% "inversion" was partly definitional. Re-grading the same Part C single-turn bundles (no re-run; `runs/kbfair_report/`) with two tool-agnostic measures: **recall source (any tool: retrieval+KB)** = `prod` **100%** vs `kb_off` 96% (n=24), and **cited-correct source/lesson** (does the *answer* cite the labeled source/lesson, resolved from both tools via `kb_manifest`) = **100%/100% for both** (n=14 corpus). Verified the any-tool credit is real, not a regex hit: in all **12** prod cases where retrieval missed, the agent had browsed that exact gold source via `run_kb_command`. So turning the KB off does **not** improve grounding — both configs find and cite the right source. `kb_off`'s genuine, non-artifact wins remain latency (19s vs 38s p50 TTFT), efficiency (3.6 vs 10.1 LLM calls/turn), and a slight single-turn quality edge (behavior 88% vs 79%, key-points 82% vs 78%) — all from doing one retrieval call instead of many browse rounds, not from better recall. **Supersedes F17's recall reading; F17's cost/memory points stand.** Caveats: n=24 screen subset, 1 trial; `cited_correct` saturates at 100% here (coarse at this n), so any-tool recall is the discriminating fair metric; bundle KB output is capped at 6000 chars (could undercount KB hits, which only biases *against* prod — the true gap can be smaller, not larger). New auto metrics in `evals/grade.py`: `recall_anytool_source`, `cited_correct_source/lesson` (free to recompute on any saved bundle). Caveat on the efficiency read: `kb_off`'s low tool-calls/turn (2.7 vs 8.7) is partly *structural* — an entire tool was removed — so it is not on the same "re-work" axis as the compaction arms' tool-call counts.

- **F24 (2026-06-19) — F3's "0 clears" was a blind-signal artifact: `cleared_tool_outputs` cannot observe `ClearToolUsesEdit`, which fires but invisibly.** `ContextEditingMiddleware.wrap_model_call` applies the clear to a `deepcopy` of the per-call message view and returns it via `request.override(messages=...)` (`langchain/agents/middleware/context_editing.py:251-255`) — it never writes the placeholder back to the checkpoint, but `telemetry.context_window_stats` counts placeholders in the *checkpointed* `state_messages` (`chat_service.py:1876`). So the counter reads **0 for every preset including `aggressive`** across all `runs/`, regardless of whether clearing fired — "0 clears" is structural, not evidence of absence. The real effect lands in input tokens: on sessions `clear_retrieval_kb` cut mean input **145.6k→120.5k (~17%, $0.229→$0.203/turn)** vs `prod` by clearing the retrieval payloads `prod` exempts, while on single-turn it **backfired to 308k/$0.477** (re-retrieval churn to recover what it cut — the `aggressive`/F19 pathology). So clearing is real and *mixed* (session win, single-turn loss), not absent. **Supersedes F3's "clearing never fires" reading; F3's structural point stands** — `prod`'s `exclude_tools=("retrieve_tutor_context",)` provably leaves the dominant retrieval payloads in place by construction, so prod-clearing still can't touch the big tokens. Caveats: the session token saving is 1-trial screen n; library clearing stays telemetry-invisible (the probe gate survives only because summarization, which *does* persist, co-fires in these arms — a clearing-only arm would false-fail `check_triggers`). Detect clearing via input-token deltas, not `cleared_tool_outputs`.

- **F25 (2026-06-19) — DeepSeek-V4-Flash replicates F9/F10 and extends them to long horizon: `full_history` is cheapest AND best-memory on every tier, so the cost crossover never appears.** A full Part-C-style v2 matrix re-run on **DeepSeek-V4-Flash (first-party API, not Gemini)** — 12 arms, 840 turns, 0 errors; Tier 1 contradiction (3×22-turn) + Tier 2 long-horizon (2×36-turn); reports `runs/ds_v2_t{1,2}_report/`, raw on HF `eval_runs/part_f_deepseek_v2`. `full_history` is the **cheapest** arm on both tiers (**$0.0069/turn** at 22t, **$0.0101/turn** at 36t), undercutting every compaction arm — while billing the *most* tokens (1.78M input tok/turn at long horizon) because ~**97% are cache-hit**. DeepSeek's ~50× cache discount ($0.14 cache-miss vs $0.0028 cache-hit input) makes the enormous *cached* prefix cheaper than any compaction's cache-breaking rewrite, so the summarization arms are priciest (`prod` $0.0336, `selective_retention` $0.0357). F9's inversion therefore holds **even at 36 turns**, and harder than on Gemini (10× discount → crossover would appear sooner; DeepSeek's 50× pushes it past every length tested). Per-turn cost is ~10-15× below Gemini on the comparable tier. **Provider caveat:** `incontext_history_retrieval` does *not* beat `full_history` on cost here ($0.0147 vs $0.0101) — the opposite of a teammate's OpenRouter run, where fallback routing fragments the prefix cache; cost comparisons must hold the provider/caching path constant (first-party single-endpoint caching verified live, `cache_read` non-zero). Caveats: thin n (Tier 1 n=3, Tier 2 n=2 per arm, 1 trial) — coarse rankings.

- **F26 (2026-06-19) — The v2 contradiction question (Q2) is answered on DeepSeek: `full_history` resolves contradictions, compaction loses them, so no temporal memory is needed.** Same matrix, judge-graded via the validated **subagent path** (28 probes, blind, `judge.py` rubric — F14). `full_history` scores **3/3 contradiction** and **2/2 long-horizon recall** — perfect memory on both tiers, including the regime where it could in theory lose (it carries both A and A′ raw and must pick the current A′). `prod` summarization **fails contradictions 0/3** (evicts the updated fact — F10 confirmed and sharpened) and 1/2 long-horizon. `incontext_history_retrieval` matches `full_history` (3/3, 2/2) by retrieving the old turns, but costs more (F25); `profile_memory` partially rescues contradictions (2/3) via its store (the F22 pattern). Among long-horizon compaction arms, `prompt_compression` is least-bad (2/2 — shrinks text, drops no facts) and `context_reset` is dominated (0/2). Validity: compaction fired at **100% of probes** for every compaction arm, **0%** for `full_history` (gate passed). **Resolves item 4's conditional — `temporal_graph_memory` is not triggered**, since `full_history` did not fail contradictions. Caveats: thin n (3/2 per arm, 1 trial); the judge is reproducibility-validated (F14), not independent-human.

- **F27 (2026-06-20) — On single-turn, `full_history`'s latency edge is a *tail* effect from `prod` summarizing mid-turn, not a median win; and `total − ttft` cannot de-confound latency.** Re-read of the Part B/C single-turn bundles (no re-run). Paired per question, the `prod` − `full_history` median **TTFT** delta is **+0.43s** (Part B, n=120, 2 trials) / +3.5s (Part C, n=24, 1 trial) and `prod` is slower in only **62/120** cases — a near-tie at the median, *not* the sessions gap (F9's 17s-vs-21–43s is multi-turn and cannot apply with no prior history). The gap lives in the **tail**: `SummarizationMiddleware` fires on **29/120** single-turn `prod` turns (24%) because the agent's *own* retrieval/browse output crosses the 30k trigger **within one turn**; those turns run **52s TTFT / 16 LLM calls** vs **29s / 8** when it doesn't, and on the *same* questions `prod` does 2–3× `full_history`'s calls (e.g. 9→27, 12→32) — the F9/F4 re-work pathology (summary drops raw evidence → re-browse) inside a single turn. `full_history` never summarizes, so it skips the tail; that lifts its *mean* (Part C 35s vs 51s) while the *median* stays tied. **Methodological corollary (extends the time-of-run caveat below):** `total − ttft` does **not** strip API-load noise — it is the answer-streaming tail (**~6.0s for both presets**; +1.5s even when summarization fires), because the entire agentic loop incl. the summarization tax lands in `ttft`, so subtracting it deletes the signal. Latency ≈ work × time-per-work, and the confound smears across ~20 calls, so no algebra on one sequential run removes it; the **unconfounded** read is the work metrics — **LLM calls/turn and tokens/turn** (run-time-independent like cost/recall, already in every bundle) — which are also the upstream cause. Live LangSmith spot-check (3 interleaved pairs, one browse-heavy single-turn Q) reproduced **both** directions: `prod` faster once (summary collapsed 268k→23k ctx, 65s) and slower twice (re-work: 23/32 calls, 116s/124s, $2.27/$1.04 vs `full_history` $0.74/$0.33), and LangSmith's independent latency matched telemetry to the ms (`prod` 65.07s vs 65.16s; `full_history` 103.64s vs 104.28s), trees confirming `full_history` carries no `SummarizationMiddleware`. Caveats: thin n and time-of-run confounded for absolute seconds; the live pairs are n=3 on one non-representative question. Same class as F24 (single-turn *clearing* backfire) on the other compaction half.

- **F28 (2026-06-19) — GraphRAG does not beat classical hybrid RAG on single-turn course Q&A: it ties on grounding accuracy and costs +44% $/turn, empirically confirming the earlier decision to drop it.** A scoped, fair head-to-head where the *only* variable is the retrieval backend behind `retrieve_tutor_context`: production hybrid `LocalChromaRetriever` (dense Cohere `embed-v4` + BM25 → RRF → Cohere rerank → token budget) vs a true Microsoft GraphRAG index (`app/graph_rag.py` `GraphRAGRetriever` — local-search-style context assembly over entities/text-units/community-reports, mapped back to real `source`/`url` via `corpus_manifest`, then the **same Cohere rerank + token budget** as classical; context-provider only, so the agent's **Gemini 3.5 Flash is the sole generation model in both arms**). 41 `full_stack_ai_engineering` single-turn cases (27 `answer_from_corpus`), Gemini 3.5 Flash, `--disable-kb` so the retriever is the only variable, `--scope-sources`, 1 trial, **0 errors/arm**. Both arms **surface and cite the right source 100%** of the time and tie on lesson recall (**76%**) and cited-correct lesson (**85%**); classical is slightly *better* at ranking the right lesson (**MRR 0.70 vs 0.65**). GraphRAG pulls community-report context, so it spends **+61% input tokens (178k vs 110k) and +44% $/turn ($0.212 vs $0.147)** and is a touch slower, with no accuracy payoff (the +3pt behavior proxy is a coarse n=37 code check, not a quality signal). So GraphRAG's community structure does not help typical single-hop course Q&A — this **empirically supports** the prior "dropped on faith" decision (see "Deliberately not pursued" below) rather than asserting it. **Reusable artifact:** the scoped index (90 docs → 15,059 entities / 32,748 relationships / 3,371 community reports, built with Gemini 2.5 Flash for **$44.96**) is published to the private HF dataset (`graphrag/output/`, pulled on demand, *not* by prod cold-start — `ensure_local_vector_db` ignores `graphrag/**`), so the eval re-runs without the ~$45 rebuild. Full methodology, build/run commands, and the indexing gotchas (route Gemini 2.5 via its OpenAI-compatible endpoint; `run_graphrag_index.sh` loops `graphrag index` until `entities.parquet` exists since one exhausted retry aborts the stage) live in **`evals/graphrag.md`**. Connects to F17/F23 (the `kb_off` grounding discussion — classical RAG already grounds well). Caveats: scoped to one source, single-turn, n=41 (27 corpus-answer), 1 trial, auto metrics only (key-point/behavior human grades pending); `recall@source` is saturated by `--scope-sources`, so lesson recall / MRR / cited-correct-lesson are the discriminating numbers. GraphRAG's theoretical edge is **multi-hop / cross-document synthesis**, which this single-hop battery barely exercises — so this says it doesn't help on single-hop course Q&A, *not* that it never helps; a fair test of that strength needs a multi-hop `_v2` probe set (out of scope here). (Branch `experiment/graphrag-vs-rag`, PR #2.)

- **F29 (2026-06-20) — Keeping a large *static document* in context is a different regime from conversational `full_history`, and it is one boundary where "keep everything" stops winning: for a single long lesson queried repeatedly, retrieving the relevant slice matches keep-it-all at ~1/13 the cost. (Boundary on F9; standalone Gemini-2.5-Flash fleet — absolute numbers not cross-comparable.)** A keep-vs-compact-vs-retrieve study on the corpus's largest lesson (~37.5k tok) loaded once in turn 0, then **15 question-turns with no tools** (the only context is what each method keeps or builds), each answer judged against the full lesson; 11 arms, Gemini 2.5 Flash, 1 trial. `rag` (fetch the relevant chunk per question) is **cheapest *and* tied-best** (60% / 3.2k tok / $0.020); `full_history` (keep the whole lesson in the prompt) is the **priciest in-context arm and not the best** (53% / 43k tok / $0.134); the summarization family cuts ~35% of tokens for a quality drop (40–53%); `incontext_history_retrieval` ties `rag` on quality but at ~13× the tokens; **`hierarchical_summarization` is the worst trade** — priciest ($0.605) and slowest (41.8s p50) for the lowest quality (33%). `graphrag` (53%, $0.045) again does not beat plain `rag` (consistent with **F28**). **Read it as a lesson, not a head-to-head:** this is *not* "RAG beats the `full_history` memory strategy" — here `full_history` only means "the whole lesson sits in the prompt," not its multi-turn-memory role. It says **F9's "keep everything wins" is about *conversation history*; for a big *static document* you could retrieve from, hoarding it in context is the priciest option and buys nothing**, and in the production tutor retrieval (Axis B) and a memory policy (Axis A) **compose**, they don't compete. Companion boundary: F30 (small context window). New reusable presets landed in `app/memory_presets.py`: `delta_summarization`, `hierarchical_summarization`. Full matrix, the F-C1–F-C6 detail, and reproduce steps in **`evals/compaction.md`**. Caveats: n=15, 1 session, 1 trial (coarse rankings, not precise rates); judge is Gemini 2.5 Flash (same family, reads the full lesson as ground truth); **separate fleet from the Part B/C 3.5-Flash runs — quality is not cross-comparable across models**. (Branch `experiment/context-compaction`, PR #5.)

- **F30 (2026-06-20) — A small context window is a second boundary where "keep everything" stops winning, and a harder one than F29: when the document physically does not fit, "shove it all" (`full_context`) is forced into truncation and is strictly *dominated*, so retrieving the relevant slice (`rag`) is not just cheaper but necessary. (Small-window companion to F29, Axis B; local-SLM fleet — absolute numbers not cross-comparable.)** "Fit one ~37.7k-token lesson into context to answer 15 questions," 7 methods × 3 Ollama SLMs (llama3.1:8b, qwen2.5:7b-instruct, qwen3:8b no-think) on an M1 Pro/16 GB at a 32k window (`--num-ctx 32768`), no prompt caching; judge stays on Gemini 2.5 Flash (reads the full lesson as ground truth — the SLM cannot hold it, and a model must never grade itself), 1 trial. `full_context` (the whole *document* placed in the prompt, `ctx['full_context'] = lesson`, stateless per question — this is **document-stuffing**, *not* the conversational `full_history` memory policy) is **truncated on 15/15 turns** (37.7k → 32,767) and **dominated**: equal-or-worse quality (67–80%) at **~12–15× the latency** (266–346s vs ~20s) — silent information loss *plus* a huge latency tax (e.g. qwen2.5 `full_context` answered "no model was mentioned" for a fact the truncated intro had named). `rag` (chunk + Cohere-embed + retrieve top-k per question) is **top or tied-top on all three** (qwen2.5 100%, qwen3 100%, llama 80%) at ~2.9k context tokens and ~20–26s. `graphrag` ties `rag` on quality but at ~2.8× tokens and ~2.5× latency — no payoff, **consistent with F28/F29**. Summarization is the floor (llama `summary` **0%**); `trim`/`selective` land mid (47–73%). This marks where F9/F15's precondition — a window big enough to hold everything, cheaply, under caching — simply fails on the hardware a workshop attendee runs. **Distinct from the conversational-memory inversion (F31):** `full_context` here is *document-stuffing* (Axis B), not `full_history`'s multi-turn role, exactly the distinction F29 drew. **Methodology fix:** the first pass scored ~64% of judge verdicts as fails because a long JSON `reason` hit `max_tokens` and truncated the verdict; hardened the judge (25-word reason cap + leading-boolean recovery) and re-graded the saved answers offline (`evals/rejudge_compaction.py`, **0 unparseable** after) — the directly-measured axes (tokens, latency, overflow) were never affected. Caveats: n=15/method, 1 trial, one lesson, one domain — read as a ranking, not rates; **separate fleet (local Ollama SLMs, no caching, Gemini-2.5-Flash judge) — quality/cost not cross-comparable** with the Part B/C 3.5-Flash, the F29 2.5-Flash, or the DeepSeek runs. Full writeup: **`evals/slm_compaction.md`** (Experiment 1). (Branch `experiment/slm-compaction`, PR #3.)
- **F31 (2026-06-20) — On an SLM, compacting a *growing conversation history* has no universal best method, and "keep everything" (`full_history`) never wins — and model capability dominates the choice of method. (SLM counterpart to F9/F15 on Axis A; same local fleet — absolute numbers not cross-comparable.)** Same lesson loaded turn 0 of a session, 15 question-turns, **retrieval off**, run through the real app middlewares (the production memory presets) on the 3 SLMs at `num_ctx=32768`, judged on Gemini against the full lesson, 1 trial. The best preset is **model-dependent** — `prompt_compression` on qwen2.5 (60%), `hierarchical_summarization` on llama3.1 (40%), `summarization_only`/`delta_summarization` on qwen3 (73%) — so there is **no single "best compaction" to recommend**. `full_history` (the genuine conversational memory policy — keep *every prior turn*, no summarization or context-editing; `summarization=False, context_editing=False`) **tops no model** (27% / 7% / 67%) and is near-bottom on the two weaker ones; `sliding_window` is reliably among the worst (13% / 13% / 40%). **Model capability dominates the method:** qwen3 (40–73% across *every* preset) >> qwen2.5 (13–60%) >> llama3.1 (0–40%) — choosing a stronger small model buys more than choosing the best strategy. **Mechanism (why keep-all can't win here):** on an SLM "keep everything" (`full_history`) can't even be truncated-and-kept — Ollama's server-side context fitting **evicts the whole oversized turn-0 lesson** from turn 1 on (the checkpoint still holds it; the model receives only ~0.3–1.4k tok of accumulated Q&A), so a method that *summarizes the lesson into a small injected message* is the only way to retain its gist; on Gemini's large window this never happens (F9's `full_history` held ~43k/turn). `hierarchical_summarization` is by far the priciest (≈30-call map-reduce, p50 233–333s, ~25–29k retained tok) for no quality lead except on the weakest model. This **inverts F9/F15's conversational-memory result** (keep-all cheapest *and* best) in the regime where the runtime physically cannot keep it. Caveats: n=15, 1 trial, LLM-judge variance moves individual cells ±1–3/15 (read only the big gaps); the `full_history` numbers are **Ollama-eviction-specific** (a different serving layer could keep a truncated lesson instead); **same separate-fleet comparability caveat as F30**. Writeup: **`evals/slm_compaction.md`** (Experiment 2). (Branch `experiment/slm-compaction`, PR #3.)
- **F32 (2026-06-22) — Synthesis: "keep everything in context" wins only under a specific precondition — a large window that holds the whole history cheaply under prompt caching — and we have now mapped three boundaries where that precondition breaks and keep-all stops winning.** **Two distinct "keep-all" arms — never conflate them:** `full_history` (Axis A — F9/F15/F31) is the *memory policy* that keeps every prior **conversation turn** with no compaction (`summarization=False, context_editing=False` in `app/memory_presets.py`); `full_context` (Axis B — F30) is **document-stuffing** — the whole *source document* placed in the prompt (`ctx['full_context'] = lesson`). Same slogan ("keep everything"), different mechanisms. **F29 is the trap to watch:** its keep-all arm ran the `full_history` *preset* but over a single static lesson parked in turn 0 (`turns = [lesson] + questions`), so what it actually measured is the `full_context` idea (a document held in the prompt), **not** `full_history`'s multi-turn-memory role — which is why F29 sits on the *document* boundary (1) below, alongside F30, and **not** the conversational boundary (3). F9/F15 established the default (raw `full_history` is cheapest *and* best memory on conversational sessions), and F2/F25/F26 explain *why* it holds and even strengthens with horizon (the big **cached** prefix is cheaper than any compaction's cache-breaking rewrite; DeepSeek's ~50× cache pushes the crossover past 36 turns). Remove a piece of that precondition and keep-all loses, along three boundaries: **(1) a large static document you could retrieve from** (F29, Axis B, large cached window) — retrieving the slice *matches* keep-all at ~1/13 the cost, so hoarding the doc is the priciest option and buys nothing; **(2) a context window too small to hold the document** (F30, Axis B, SLM) — keep-all (`full_context`) is forced into truncation and is strictly dominated, so retrieval is *necessary*, not merely cheaper; **(3) a growing history on a small window** (F31, Axis A, SLM) — the runtime *evicts* the oversized turn, so keep-all (`full_history`) never tops the table and some compaction is mandatory, though no single method wins. Two cross-cutting lessons fall out: **model capability dominates the compaction method** on weak local models (F30/F31 — a better small model beats a better strategy), and the **axes compose, they don't compete** (F29) — retrieval (Axis B, *how to fit a document*) and a memory policy (Axis A, *how to compact history*) are orthogonal, so "RAG vs `full_history`" is a category error. **Net guidance:** keep-everything stays the right default in the production regime (large cached cloud window + conversational history; F9/F15/F25/F26); reach for retrieval or compaction exactly when one precondition breaks — a big static doc, a small window, or no caching. **Comparability:** this ties the *lessons* across fleets (3.5-Flash screen, 2.5-Flash F29, DeepSeek F25/F26, local SLMs F30/F31), **not** their absolute numbers, which are not cross-comparable.

- **F33 (2026-06-23) — New quality axis (holistic "would staff send this?"): on single-turn, compaction degrades *answer quality itself*, and the degradation dose-responds to how often summarization fires *within a single turn*. (Independent quality-axis confirmation of F4/F19/F27.)** Added two judge rubrics in `evals/judge.py`: **holistic** (whole-answer staff-approval / does-it-help-the-student-learn gate, emitted for every answered turn) and **faithfulness** (groundedness vs retrieved evidence — *parked*, see the correction below). Graded blind by the validated subagent path (F14) across all single-turn runs; holistic rows added to `evals/report.py`, tagged `[judge]`. The single-turn premise is that memory presets should barely differ ("Memory presets should score about the same here; big differences suggest a bug", §The dataset) — yet holistic spans **57%→100%**, and it is *not* a bug: it tracks within-turn summarization. Dose-response: arms that never summarize mid-turn score 92–100% (`full_history` 92% Part C / 92% Part B n=120; `kb_off` **100%** at just 3.6 LLM calls; `sliding_window`/`incontext_history_retrieval`/`selective_retention`/`retrieval_budget_30k` 96%), while arms whose *own* retrieval/KB output trips the trigger mid-turn sink in trigger order — `observation_truncation` **71%** (22.8 LLM calls/turn, 12/24 turns summarized), `context_reset` **62%** (18/24), `aggressive` **57%** (Part B n=120, **110/120** turns summarized mid-turn). So the F4/F19/F27 within-turn re-work pathology is not merely slower/pricier — it makes the *answers worse*, at a product-relevant magnitude (`aggressive`: ~half its single-turn answers would not be sent). Caveats: Part C n=24 (directional); the firm cell is Part B `aggressive` 57% vs `full_history` 92% at n=120; judge reproducibility-validated on probes (F14), **not** independently on holistic.

- **F34 (2026-06-23) — Across a session, standard compaction's damage is *surgical*: it collapses memory while answer quality stays ~100% — but that "quality" is partly an artifact of a history-blind judge, so "looks fine" ≠ "served the student". Only the most aggressive rewrite (`context_reset`) degrades the answers themselves, progressively over the session. (Refines F10/F12 on the holistic axis.)** Holistic graded on every answered turn of `b_se`+`c_se` (683 turns, 15 arms, blind subagent path). **The decoupling:** `prod` probe **58%**(c)/**38%**(b) yet holistic **97/99%**; `kb_off` probe 33% / holistic 100%; `selective_retention` probe 25% / holistic 100%; `profile_memory` probe 38% / holistic 100% — memory and answer-quality are *independent failure axes*. The lone exception is `context_reset` (probe 17%, holistic **69%**, holistic dropping **−24% early→late turn**; `sliding_window` a milder −16% late): broad, progressive quality degradation appears only under the most destructive history rewriting, otherwise quality holds flat (`Δ late−early` ≈ 0 for all standard arms). **But "high holistic" is misleading on exactly the turns that matter** — holistic is graded *blind to the session history* (sessions carry no per-turn gold answer, and the judge sees only question+answer), so on the **memory-failed probe turns** it still rated the (generic, context-ignoring) answer "good" **100% of the time** for every standard arm (`c_se_prod` 5/5, `b_se_prod` 15/15, `kb_off` 8/8, `selective_retention` 9/9; `forgot+looks-bad` = **0**) — only `context_reset` forgot so badly the blind judge caught it (6/10 forgotten-fact turns also looked bad). The failure mode is therefore *confidently generic*: a well-formed answer that silently ignores what the student stated earlier, which only the targeted probe detects. **Lesson:** holistic (context-free quality) and probe (context-aware memory) measure different things and must be read together — "answer quality looks fine" does **not** mean the student was served. This refines F10 (memory dies but general answer quality survives) and F12 (only the *aggressive* rewrites *also* degrade quality). Caveats: `c_se` is 3 sessions × 1 trial (~18 turns/early-late bucket), `b_se` ×2 trials; session holistic graded without a staff reference (the within-arm early/late comparison controls for that); judge reproducibility-validated on probes (F14), not on holistic — the blind-spot above is itself why a held-out human validation of holistic is the next gate.

- **F35 (2026-07-15) — 2×2 factorial on DeepSeek: the cheap win is a *stable* tool-output cap, not compaction — capping cuts cost −38% at no measured quality/memory loss while preserving the cache, and summarization adds cost on BOTH sides of the factorial, so it still doesn't pay for itself even after capping. (Decomposes F25; Axis A × Axis B; DeepSeek-V4-Flash first-party — absolute numbers not cross-comparable with the Gemini fleets.)** *Experiment: **DeepSeek long-context compaction study, stage 1.** The study asks whether summarization-based history compaction ever pays for itself on the production tutor running its default model (DeepSeek-V4-Flash, first-party API) over long tutoring sessions, once prompt caching and tool-output size are accounted for. F25 answered "no, `full_history` stays cheapest through 36 turns" — but its compaction arms confounded two mechanisms, the history policy and the giant raw tool outputs that dominate input tokens (F1), so it couldn't say* where *the money actually goes. Stage 1 is the clean decomposition: a 2×2 factorial crossing the history policy (keep everything vs summarize at a 200k trigger, retaining a 50k recent tail) with tool-output handling (raw vs a stable 10k-token cap applied once when the output enters history), giving four arms — `exp_fh_raw` / `exp_fh_cap10k` / `exp_c200_raw` / `exp_c200_cap10k` (`app/memory_presets.py`) — whose paired contrasts isolate each mechanism's cost/memory effect and their interaction. Stage 2 (trigger-threshold sensitivity: `exp_c400_cap10k` / `exp_c800_cap10k`) is built but not yet run. Method: battery `battery_sessions_v2` (22-/36-turn scripted tutoring sessions that plant student facts early and probe them late), 3 trials, run via the `evals.run_compaction_experiment` paired lockstep runner (all four arms advance turn-by-turn together in randomized within-turn order, each arm/session/trial under a distinct DeepSeek cache `user_id` so no arm warms another's KV cache; immutable run fingerprint; `evals/contributing.md` §2). "triggerfix" in the run name marks this as the corrected rerun after the earlier stage-1 attempt's summarization-trigger bug. Run `runs/deepseek_compaction_stage1_triggerfix_20260715`; full writeup `runs/deepseek_compaction_stage1_triggerfix_20260715/report/stage1_findings.md`; raw on HF `eval_runs/deepseek_compaction_stage1/`.* Four arms {`full_history`, summarize@200k keep-50k} × {raw tool outputs, stable 10k-token cap (40k bytes) applied once when the output enters history} on sessions-v2 Tier-1 contradiction (3×22t) + Tier-2 long-horizon (2×36t), ×3 trials, run turn-level-lockstep with randomized within-turn order and a distinct DeepSeek cache `user_id` per arm/session/trial (no cross-arm KV warming); 414 turns/arm, 0 errors. The earlier stage-1 trigger bug is fixed and evidenced: every compaction fired at ≥200,056 pre-compaction tokens with 92k–198k-token summary inputs, and the `full_history` arms compacted 0 times. Cost (mean $/trajectory): capping alone **$0.189 → $0.117 (−38%, cheaper in 14/15 paired trajectories)** with the cache-hit ratio intact (96.0%→95.9%) — it removes ~1MB of repeated tool payload per trajectory *without* rewriting the prefix, so it composes with caching instead of fighting it (the F1→F24/F25 thread: the tokens live in tool outputs, and cutting them *stably* is nearly free). Summarization is a net cost **add** on both sides: +50% vs raw full-history (14/15) and **+32% vs capped full-history (12/15)**; capping shrinks the compaction penalty (interaction −$0.057) but does not reverse it. Quality holds where it should: holistic 96–98% for all four arms, and the cap costs no memory (probe 87%, identical to raw full-history). Contrast with F19: per-call-view head/tail truncation churned (re-calls, 33% memory); this checkpoint-level insertion-time cap shows no churn at all (3.0 vs 3.1 LLM calls/turn, 87% memory) — though the arms differ in retained size and model too, the stability of what the model re-reads looks like the operative difference. Caveats: probes n=15/arm and only 5 session templates (the writeup's cluster-bootstrap intervals are descriptive, not population estimates); holistic is judge-graded, not human-validated (F34); DeepSeek's ~50× cache discount is load-bearing for the ranking (F25's provider caveat).

- **F36 (2026-07-15) — Where compaction hurts memory, the damage tracks the *number of lossy rewrites*, not the size of the live context — and raw tool outputs double that number, so Axis-B bloat *amplifies* Axis-A damage. "Too many tokens in the answering call" is not the failure mode: `full_history` recalls perfectly from up to 879k-token contexts. (Refines F10/F26's mechanism.)** *Experiment: same DeepSeek stage-1 compaction factorial as F35 (run `runs/deepseek_compaction_stage1_triggerfix_20260715`, battery `battery_sessions_v2`; probe-level evidence in the same `stage1_findings.md` writeup) — this is the memory half of that run's results. The battery's two tiers each target one memory failure mode: Tier 2 long-horizon (2 sessions × 36 turns × 3 trials) plants student facts/constraints early and probes them ~30 turns later, after compaction has had every chance to evict them; Tier 1 contradiction (3 × 22t × 3) plants a fact, updates it mid-session, and probes whether the agent uses the current value. Probes judge-graded via the F14-validated blinded subagent path.* Long-horizon recall (plant facts early in a 36-turn session, probe at the end): the extended 2026-07-15 audit found one of the two Tier-2 sessions (`v2_t2_fullstack_long_persona_36t`) carries recycled "project pivot" filler that outright contradicts the probed PHI constraint, so its cells cannot separate eviction-by-compression from eviction-by-believed-pivot and are excluded (repaired in `battery_sessions_v2_1.jsonl`). On the clean Tier-2 session (`v2_t2_agentic_long_persona_36t`, whose off-persona filler is orthogonal to its probed governance facts; n=3/arm): **0 summarization events → 3/3 and 3/3** (both full-history arms), **~2–3 events → 3/3 and 3/3** (`exp_c200_cap10k` and F37's structured arm), **~5–6 events → 1/3** (`exp_c200_raw`). The dose mechanism survives in coarser form — only the high-dose raw arm measurably loses far-back facts, and low-dose capped compaction is indistinguishable from full history at this n — but the fine "6/6 → 5/6 → 3/6" gradient quoted before the audit mixed in the compromised session and should not be cited. The raw arm compacts twice as often because uncapped tool payloads regrow the history to the 200k trigger faster; each event then compresses ~150k tokens into ~1.7k (~90×), and on the clean session the judge's fail reason is genuine eviction ("explicitly denies knowing the student's constraints and offers only a generic list of common governance options"). Read-time context size points the *other* way: at probe time the compaction arms answered from 95–195k-token contexts and failed, while `full_history` answered from 363–879k and never missed — so the "context rot" here lives in the summarization pipeline (a planted fact that is a tiny fraction of a payload-dominated span must survive every ~90× rewrite), not in the answering model's attention. Separately, the contradiction tier: all 11 contradiction misses across all arms sit in **one session** (`python_colab_to_local_22t`), and the miss mode is *hedging* (recap both environments, ask "which machine are you on?") rather than stale-fact use — but the 2026-07-15 battery audit found that session **confounded**, so treat its cells as unmeasured rather than as a finding: its recycled filler injects a first-person *third* environment ("my iMac", turn 16) *after* the turn-7 update and *before* the turn-21 probe, which makes the hedge a defensible reading of genuinely contradictory history, not a demonstrable memory failure. The other two contradiction sessions are confound-free (audited: zero post-update first-person environment claims) and had **zero misses in any arm** — so this run shows no contradiction gap between arms and leaves F26 standing; whether `full_history` hedges when it truly holds A and A′ needs the repaired session rerun (fix as a new battery version — v1/v2 files are frozen). Caveats: the clean Tier-2 evidence is n=3/arm — read "high dose fails, low dose doesn't", nothing finer; Tier-1 does not guarantee eviction before its probe (all five of `exp_c200_cap10k`'s compaction-dormant trajectories are Tier-1, so in capped arms Tier-1 tests update-adherence, not contradiction-under-eviction); Tier-2's probed constraints are partially guessable from world knowledge (a governance rule for a procurement agent), which inflates every arm's absolute recall equally and biases *against* the dose signal. Aggregate probe accuracy restricted to audit-clean probes (9/arm): full-history arms **9/9 and 9/9**, capped compaction **9/9** (XML) / **9/9** (structured), raw compaction **7/9**.

- **F37 (2026-07-15) — A chunk of "compaction's cost" was an implementation artifact, not physics: LangChain's stock summarizer breaks the provider cache on the summary-generation call itself, and a prefix-preserving compaction request removes that — summarizer cache hit 0%→94%, summarization cost −87%, total −14%/trajectory vs the stock arm, with equal-or-better memory. But the *post-compaction* prefix break is structural, so compaction still costs +14% vs capped full history: F35's ranking stands, with a smaller gap. (Same regime as F35/F36 — DeepSeek-V4-Flash first-party, absolute numbers not cross-comparable with the Gemini fleets.)** *Experiment: **DeepSeek compaction study, stage 1 follow-up — prefix-preserving ("structured") compaction.** The stock `SummarizationMiddleware` serializes the selected messages into a fresh XML prompt to ask for the summary, so even summary *generation* pays 100% cache-miss prices on ~150k-token inputs; `PrefixPreservingCompactionMiddleware` (`app/chat_service.py`, `summarization_strategy="structured_prefix"`) instead sends the *unchanged current request prefix* plus one checkpoint instruction, binding the same model/tools/settings, so the summarizer call rides the same cache as the agent. Single arm `exp_c200_cap10k_structured` — identical to `exp_c200_cap10k` except the strategy — run `runs/deepseek_structured_only_stage1_20260715` on the same `battery_sessions_v2` × 3 trials (414 turns, 0 errors), compared pairwise per session×trial against the four stage-1 arms; graded through the same blinded-subagent pipeline (429 judgments, 0 missing). Writeup `runs/deepseek_structured_only_stage1_20260715/report/structured_findings.md`; raw on HF `eval_runs/deepseek_compaction_stage1/`. A 2026-07-15 source-level check of the open-source Codex CLI confirmed this strategy is a faithful analog of Codex's local compaction — unchanged history + appended instruction, explicitly cache-preserving — while Codex's OpenAI-default *remote v2* path is the provider-native opaque-compaction primitive a client middleware cannot replicate; comparison + adoption ideas in `runs/deepseek_structured_only_stage1_20260715/report/codex_comparison.md`.* Cost: vs the stock XML arm, **−$0.0217/trajectory (−14.0%, cheaper in 11/15 pairs) at identical compaction dose** (1.27 events/trajectory both): the summarizer call's cache-hit ratio goes **0.0% → 94.2%**, so summarization cost falls $0.0276 → $0.0036/trajectory (−87%) — even though the structured arm *bills more* raw input (10.6M vs 9.9M tokens/trajectory; the checkpoint request re-reads the whole prefix at the ~50× cache-read discount). Tokens ≠ dollars, again (F2/F25). vs capped full history it remains **+13.7% (cheaper in only 4/15)**: what's left is the structural boundary — installing the summary necessarily rewrites the *next* agent call's prefix, which no client-side middleware can avoid (that would take a provider-native compaction/continuation primitive) — plus the extra agent work (3.5 vs 3.1 LLM calls/turn). Memory: equal-best of the compaction arms — on audit-clean probes (excluding the two battery-compromised sessions, see F36) it scores **9/9**, tied with both full-history arms and capped-XML compaction, while raw compaction scores 7/9; the pre-audit "6/6 vs 5/6" long-horizon edge over the XML arm rode entirely on the compromised fullstack session and should not be cited — whether the intact-prefix summary *also* preserves facts better than the XML rewrite is unmeasured at this n. Holistic flat at 96%. Its single overall probe miss is a contradiction hedge in `python_colab_to_local_22t` — the confounded session — a battery artifact, not an arm signal. Trigger gate: compaction fired in 12/15 trajectories (19 events, min pre-compaction observation 200,053 tokens; the 3 dormant trajectories all Tier-1, and 100% of Tier-2 probes ran under compaction). Caveats: the memory edge is 1–2 probes at n=6 long-horizon / n=15 total — directional; the arm ran solo, not turn-level-interleaved with the stage-1 arms (token/cost metrics are run-time-independent, but treat its latency rows as confounded, F27); holistic judge not human-validated (F34); DeepSeek's ~50× cache discount is load-bearing (F25's provider caveat).

- **F38 (2026-07-15) — The keep-everything-vs-compact crossover is set by one number: the cached-input price. Repricing our stage-1 traces call-by-call shows "full history wins" is a fact about DeepSeek's pricing, not a law: at GPT-5.6 Sol's public rates the *same recorded workload* already flips to favor compaction at 36 turns — quantitatively consistent with why OpenAI's Codex auto-compacts at ~272k while our tutor, on DeepSeek, should not compact at all. (Analysis-only finding — no new run; adds the missing fourth boundary to F32's list.)** *Experiment: same-trace repricing over the stage-1 arms (runs `runs/deepseek_compaction_stage1_triggerfix_20260715` + `runs/deepseek_structured_only_stage1_20260715`, F35/F37): every recorded model call's (fresh, cached, output) token triple re-costed at GPT-5.6 Sol's public API rates ($5 / $0.50 / $30 per M — a 10× cache discount vs DeepSeek's ~50×), with and without the API's >272k long-context cliff (2× input / 1.5× output on the whole request; per OpenAI's Tibo, the cliff is NOT charged on Codex subscriptions — the stated real driver is accumulated cache-read cost scaling with the live window re-read on every tool call).* Result: `exp_fh_cap10k` vs `exp_c200_cap10k_structured` goes from **$0.117 vs $0.134/trajectory (full history −13%) at DeepSeek prices** to **$9.85 vs $9.94 (≈tied) at GPT rates** — and the tie hides a horizon split: full history still wins the 22-turn tier ($6.29 vs $6.76) but **already loses the 36-turn tier ($15.18 vs $14.72), no pricing cliff involved** (directional: 3/6 tier-2 pairs, mean −$0.46; the API cliff widens it to $12.56 vs $9.94 overall — 176/1,247 full-history calls exceed 272k by provider-reported input tokens, the count the dollar figure uses; an earlier draft said 191, which counted by the app's approximate context estimate). Cache reads are **60%** of full history's repriced spend (vs ~28% at DeepSeek prices) — exactly OpenAI's named mechanism, visible in our own traces. Closed form at this 22/36-turn horizon: full history wins iff cached input costs < **~$0.55/M** — DeepSeek charges $0.0028 (200× under the line), GPT-5.6 Sol $0.50 (right at it), so the crossover turn falls from "beyond everything we tested" to "inside ordinary sessions" purely by the price constant. (2026-07-16 refinement: $0.55/M is a mix-specific aggregate over the 9-short/6-long trajectory mix — the 22-turn subset alone ties at ~$2.91/M and the 36-turn subset at ~$0.39/M; full sensitivity analysis, token receipts, and the GPT-5.6 cache-write caveat in `runs/deepseek_structured_only_stage1_20260715/report/gpt56_repricing_counterfactual.md`.) Reading: F9/F15/F25/F35's "keep everything wins" carries an implicit precondition F32's list omitted — **a deep-enough cache discount** — the fourth boundary alongside F32's three (static doc, small window, no caching). It rationalizes Codex's design (compact once cache-read accumulation crosses the compaction tax; not earlier than quality forces, since compaction damage is dose-responsive — F36 — matching Codex's own repeated-compaction accuracy warning; and don't buy bigger windows for quality — flat above 272k per OpenAI, consistent with F6/F26/F35 where 880k-token full-history contexts stayed perfectly accurate) but does **not** derive the specific 272k, which is OpenAI-specific (GPT-5.5-lineage standard-context serving boundary; the API cliff sits exactly there; a Codex 372k trial was reverted for usage cost). Caveats: same-trajectory repricing assumes GPT-5.6 would reproduce DeepSeek's call pattern and ~96% cache-hit rate (imperfect caching hurts full history *more* — F25's routing note — so the real crossover is likely earlier, not later); excludes cache-write billing on both sides; subscription "usage" is OpenAI-internal accounting, not API dollars; trigger sensitivity is unrun (`exp_c400_cap10k`/`exp_c800_cap10k` built, pending), so the optimal-trigger curve is unknown even on DeepSeek.

**Screen winners → promotion candidates:** `incontext_history_retrieval` (needs the long-session test), `retrieval_budget_30k` and `clear_retrieval_kb` (cheap Axis-B wins; clear_retrieval_kb is cheaper than prod with similar memory + better recall/key-points), anchored by `full_history`. **Drop:** `context_reset`, `observation_truncation`, `selective_retention`.

**Part C — what remains (beyond the workshop).** The screen answered the core question; everything below *hardens* or *extends* it and is **gated on a spend or data decision from Omar** — none is required for the workshop.

1. **Promotion (rigor, not new findings).** Re-run the 2-3 winners (+ 1-2 combos) on the full batteries (60 single-turn + 32 sessions + 30 replay) × 3 trials, with paired statistical tests and per-variant failure-taxonomy diffs, to turn the screen's coarse n=12 rankings into confident numbers; optionally an Anthropic (Haiku) re-run of the prefix-rewriting arms (`prompt_compression`, `context_reset`, `aggressive`) to test whether explicit caching flips F2's cost ranking. **~$1,300-1,800 at corrected pricing** (the old "$300-400" was at pre-correction prices; × the 4.4 fix — estimate precisely before running, since the per-turn cost is now ~$0.25).
2. **Finish the v2 contradiction tier — the highest-value new dataset (already built and partially run; see the V2 execution note below).** Moderate sessions: plant A, update to A′, then probe after prod compaction has evicted A from the kept-recent window. This is the one correctness regime where `full_history` itself might lose (it sees both A and A′ and must choose the current one) — yet the partial run so far shows the opposite (`full_history` contradiction **2/2** vs `prod` **0/3**), so that regime has not actually appeared. What remains is finishing Tier 1 (the missing `full_history` session, then Tier-1 `profile_memory` and optionally `incontext_history_retrieval`) and adding trials, not building the dataset. **(2026-06-19 — now run in full on DeepSeek-V4-Flash rather than Gemini: F26 confirms `full_history` 3/3 contradiction vs `prod` 0/3, so the regime did not appear on a second model either; the Gemini Tier-1 finish is now optional cross-model confirmation.)**
3. **The `incontext_history_retrieval` long-session cost test — principled but less product-critical.** The screen showed `incontext` works (83%, F16) but couldn't show a cost benefit on ≤13-turn sessions. Whether retrieve-old-turns beats `full_history` *once sessions get genuinely long* needs a token-calibrated long-session v2 tier; useful for the workshop argument, but gated by whether that regime matters in real tutor telemetry. **(2026-06-19 — run on DeepSeek (F25) at 36 turns: `full_history` stays cheapest and `incontext` works but does not beat it on cost under first-party caching; whether the answer flips on a 10× cache provider like Gemini/Anthropic is still open.)**
4. **Conditional builds.** Build `temporal_graph_memory` only if `full_history` fails contradictions (F26: it did **not** on DeepSeek — 3/3 — so this stays unbuilt), `delta_summarization` only if the long-horizon cost tier shows room to beat full history (F25: `full_history` still cheapest at 36t on DeepSeek, so no room there — though a 10× cache provider is untested), and `entity_memory` only if multi-project probes show fact bleed. `collection_memory` is a separate cheap v1/product follow-up for verbatim fact storage (F22 tested the current profile write-back, not that). `hierarchical_summarization` and `sleeptime_consolidation` are deferred. The step-by-step v2 plan is in `evals/part_c_plan.md` → "Battery v2".

**Deliberately not pursued.** *Stretch (heavier, likely later):* sub-agent isolation, reframed as a tool-output-token play — keep noisy `run_kb_command` output out of the main context (ties to F1); temporal-graph memory is already the conditional build in item 4 above. *Dropped as low-information given the findings:* the skills / lazy-prompt-loading family (only a ~458-token win — see the talk-outline correction below), procedural memory, GraphRAG (now **empirically tested** rather than dropped on faith — **F28**: ties classical RAG on grounding, costs +44% $/turn, no payoff on single-hop Q&A), and multi-agent / parallel-research agents. Context for the `kb_off` arm (F17/F23): the KB is the dominant grounding tool — ≈89% of Part B turns, ~7.7 KB vs ~0.9 retrieval calls/turn over 1,088 turns — so disabling it forces the 100k-token retrieval fallback, which is why it raised cost/latency.

**V2 execution note (2026-06-16).** A private/gitignored 6-session `battery_sessions_v2.jsonl` was built and partially run. The full v2 screen is too slow under the lower-tier Gemini key: `full_history` at concurrency 3 hit the 3M input-tokens/minute quota, and concurrency 1 works but makes the full 4-arm screen a multi-hour job. Current local artifacts (`runs/e2_v2_partial_report/report.md`) are directional, not final: `prod` completed all 6 sessions with 0 errors, all probes under compaction, **14% probe accuracy (1/7)** and contradiction **0/3**; `full_history` completed 2 Tier-1 contradiction sessions with 0 errors, no compaction, contradiction **2/2**. Priority is now **Tier 1 only**: finish the missing `full_history` contradiction session, then run Tier-1 `profile_memory` and optionally Tier-1 `incontext_history_retrieval`. **(2026-07-15: superseded — the stage-1 factorial ran the full battery ×3 trials on DeepSeek (F35–F37), and the contradiction cells here sit on the session the audit later found confounded; new runs must use `battery_sessions_v2_1.jsonl`.)**

Harness corrections (bugs in our measurement, not findings): the overnight 06-12 stall was machine sleep hanging API streams (all four pipelines stopped the same minute; fixed with the per-turn timeout); battery lesson-URLs carried a `/discussions/` suffix that silently zeroed recall@lesson until normalized (caught by the A3 smoke, re-graded from bundles without re-running).

- **Pricing correction (2026-06-13).** `MODEL_PRICING` under-priced `gemini-3.5-flash` before this date (~$0.30/$2.50 per MTok vs the correct **$1.50 input / $9.00 output / $0.15 cache-read**, verified against Google's price sheet). Every dollar figure generated earlier — the Part B table and the inline costs in findings **F2/F8/F9/F12** — is **~4.4× too low**; token counts are unaffected. Part B bundles were re-costed from their saved token counts (2026-06-15) and `runs/b_report/report.md` regenerated at correct pricing: **relative rankings are unchanged** (F9 still has `full_history` cheapest, $0.11/turn sessions), only absolute dollars move. Real Part A+B spend ≈ **$338**, not ~$73; the comprehensive program ≈ **$590** across all Gemini 3.5 Flash runs (Part B ~$323 · Part C ~$163 · rest ~$103). Use the regenerated report for absolute costs; finding dollars above are pre-correction.
- **Stable `sheet_row_id` (2026-06-15).** `evals.grade` built `sheet_row_id` with builtin `hash()`, which is salted per process, so regenerating a `handgrade_sheet.csv` produced ids that no longer matched the frozen workbook keymap — silently emptying `handgrade_workbook merge`. Switched to a `hashlib.md5` hash; the Part B re-merge was rebuilt via the deterministic `run_id` to recover the human grades.
- **`cleared_tool_outputs` is blind to library clearing (2026-06-19).** `ContextEditingMiddleware` edits a per-call `deepcopy` and never persists the placeholder to the checkpoint that `context_window_stats` reads, so the signal is 0 for all presets whether or not `ClearToolUsesEdit` fired (full detail + token evidence in **F24**). Detect clearing via input-token deltas; the custom Part C view-mechanisms avoid this by reporting through the `app.telemetry` turn-signal registry.

- **Faithfulness parked; bundle tool-output cap raised 6k→40k (2026-06-23).** The new **faithfulness** rubric (groundedness vs retrieved evidence, F33) is *not reportable on existing runs*: `evals/run_battery.py` capped each saved tool output at `TOOL_OUTPUT_MAX_CHARS = 6_000` while the KB shell returns up to **40k** (`app/kb_shell.py` `DEFAULT_MAX_OUTPUT_CHARS`), so on KB-heavy arms the judge saw a fraction of what the agent actually grounded on and scored *capture-completeness, not grounding* — faithfulness tracked the KB:retrieval ratio almost monotonically (`b_st_prod` kb:ret 10.3 → **4%**; `c_st_prod` 7.7 → 25%; `kb_off`/`graphrag` kb:ret 0 → 50–76%), the same blind-signal class as **F23/F24**. Raised the cap to **40k** (captures what the agent saw; `output_chars` still records the true length so any overflow stays visible) so *future* re-records make faithfulness measurable; existing bundles cannot be backfilled (content past 6k is gone — needs a re-run). Until then `evals/grade.py` emits faithfulness **only when evidence was captured in full** (`evidence_is_complete`, self-re-enabling after the cap raise), and the column is dropped downstream (`grading_merge.DROP_ITEM_TYPES`; `report.py` omits it directly). Holistic (F33/F34) is unaffected — it grades the answer, not the evidence.
- **Report mislabeled judge grades as human (2026-06-19, fixed).** `evals.report` hardcoded "(human grade)" on the behavior / key-point / probe-accuracy rows, but Part C runs were filled by the LLM judge (`judge_filled.csv`), not a person — so `runs/c_report/report.md` asserted a human label over judge numbers. Fix: `evals.grade` now records a `grade_source` per merged grade (the judge prefixes its note `[judge…]`; humans do not), and `evals.report` tags each column `[judge]`/`[human]`/`[mixed]` and renames the rows to "(graded)". Numbers are unchanged (and judge≈human per F14); only the provenance label is now honest. Same label-vs-measurement class as F23/F24. All Part C arms were re-graded and `c_report` regenerated.
- **`input tok/turn` is billed-across-calls, not context size (2026-06-19, clarified).** Telemetry sums `usage_metadata` over every internal model call in a turn, so a high-loop arm (`observation_truncation`, 22.8 calls) bills lots of input from small contexts while a big-payload arm (`kb_off`) bills similar totals from few large ones. Correct for *cost*, but it is not "how big is the context window." The report now shows both: **input tok/turn (billed, all calls)** and a new **context tokens/turn (window size)** row (from `context_tokens_approx`, always computed but previously unsurfaced). On single-turn the two even rank arms differently (e.g. `kb_off` has the *largest* window at 73k from its 100k retrieval payloads, but mid-pack billed input). Read F1/F6's "~200k input" as billed volume, not window size.
- **Latency rows are confounded by time-of-run (caveat, not yet fixed).** `run_part_c_screen.sh` runs arms sequentially, and TTFT/total-ms depend on API load at that moment, so cross-arm latency deltas (e.g. F19's 430s p95, F23's 19s-vs-38s) partly reflect *when* each arm ran. Token/cost/recall are run-time-independent and unaffected. Any latency claim that drives a decision should come from arms run interleaved (or repeated trials), not a single sequential screen. **`total − ttft` does *not* remove this** — it is only the answer-streaming tail (~6s, preset-insensitive); the agentic latency incl. the summarization tax is all in `ttft`, so subtracting it deletes the signal. Report the unconfounded work metrics (LLM calls/turn, tokens/turn) as the headline and treat seconds as supporting. See **F27**.
- **Latency de-confounding tooling (2026-06-22).** Acting on F27: (1) `runs/run_part_c_screen_interleaved.sh` runs the screen question-outer / arm-inner (writes `runs/ci_*`, not `runs/c_*`) so every question's arms share one API-load window — same total runs as the sequential screen, so interleaving is free. (2) `context_stats` now also emits `time_to_first_token_ms` (first reasoning/tool/text token, always ≤ `ttft_ms`), separating raw model responsiveness from the tool-call loop that `ttft_ms` includes; it flows through `evals/grade.py` to the report and shows `—` on runs recorded before it existed. (3) `evals/report.py` now leads with the run-time-independent work metrics (LLM/tool calls/turn, tokens/turn) and tags every latency row `[confounded]`, with a runtime note. All `runs/*report*` were regenerated (layout/labels only; numbers unchanged). The scripts live in gitignored `runs/` like the rest of the run tooling.
- **Mixed-judge grading confound (2026-07-15, fixed).** While grading the DeepSeek stage-1 compaction factorial (F35/F36, `runs/deepseek_compaction_stage1_triggerfix_20260715`), the first quality merge silently mixed two judges: an interrupted Codex (`gpt-5.6-sol`) subscription pass (`evals.run_subscription_grading`) had graded chunks 000–011 of the grading plan (`runs/_grading/deepseek_compaction_stage1_full_20260715`) — almost exactly the `exp_c200_cap10k` rows — before the blinded subagent workflow (`evals/grade_workflow.js`) graded the rest. The GPT judge is one-directionally harsher on holistic: on the 462 double-graded rows, 67% agreement, with GPT failing 151 rows the subagent judge passed and zero the reverse (fail rates 35% vs 3%; its extra fails are mostly technical-accuracy critiques the blind rubric can't adjudicate without evidence). That made `exp_c200_cap10k`'s holistic read 63% vs 96–98% for the other arms — a grader artifact, not an arm effect (the same rows re-graded by the subagent judge score 98%; the tell was the straddling chunk, where the *same arm* scored 76% under GPT and 97% under the subagent judge). Fix: chunks 000–011 re-graded so all 44 chunks share the F14-validated subagent judge; the Codex verdicts are preserved for comparison (`runs/_grading/deepseek_compaction_stage1_full_20260715/codex_partial/`). Lessons: **one judge per battery** — grade provenance is part of the measurement (same class as the judge-as-human label bug above); and two frontier judges disagreeing this much, one-directionally, on holistic is further evidence that holistic needs the held-out human validation F34 called for.
- **`battery_sessions_v2` filler recycling is systematically confounded (2026-07-15, two audit passes; repaired as `battery_sessions_v2_1.jsonl`).** v2's filler was recycled verbatim from v1 turns, and the recycling blindly included v1 *persona/fact-update/project-pivot* turns — each a first-person state claim — dropped into a different persona: 7 such turns across 4 of the 6 sessions. Two sessions are unusable as authored: `v2_t1_python_colab_to_local_22t` (a "my iMac" turn injects a third environment between the update and the open-ended contradiction probe — every stage-1 contradiction miss in every arm sits here) and `v2_t2_fullstack_long_persona_36t` (two "project pivot" turns relabel the student's project to hosted architectures that contradict the probed no-PHI constraint, so its long-horizon cells measure the injected contradiction, not memory); two more are muddied (`v2_t2_agentic_long_persona_36t`, `v2_t3_fullstack_two_projects_20t` — injected claims orthogonal to the probed facts). F36/F37 were corrected to audit-clean probe subsets. The same audit cleared everything else: `battery_sessions_v1` (all 32 sessions — no confounds; the update/pivot turns there are the *intended* probed facts), `battery_singleturn_v1`, and `replay_n1_v1` all clean and matching their README counts. Separate defect: 5 of `battery_personas_v1`'s authored anti-pattern checks are negation-insensitive substrings that false-fail correct answers (e.g. anti `import openai` matches the official Node SDK's `import OpenAI from 'openai'`; anti `install Python` matches "you don't need to install Python") — an unquantified downward bias on Part B's persona auto-check rows (F8/F11); repair queued. Fixes shipped: `battery_sessions_v2_1.jsonl` replaces the 7 recycled turns with neutral same-course corpus questions (probes/plants byte-identical), and the runners now **refuse `battery_sessions_v2.jsonl`** for new runs (`--allow-deprecated-battery` reproduces historical runs only). Standing lesson: recycled real-student filler must be scanned for first-person state claims against the planted facts before a battery freezes.
- **Pattern worth naming — most of our measurement bugs are telemetry/label blind-spots, not wrong math.** Six now share one shape: a number or label asserted a property the instrumentation could not observe for that config — **F21** (mechanism dormant: `profile_memory` never engaged on sessions), **F23** (recall blind to KB browsing), **F24** (clearing invisible to the checkpoint signal), the judge-as-human label above, **faithfulness graded on truncated evidence** (the 6k cap above — the judge scored what it could *see*, not what the agent *saw*), and **F34's holistic-on-session blind spot** (the quality judge sees no conversation history, so it credits answers that silently ignored the student's earlier-stated facts — "looks good" on 100% of memory-failed turns). `check_triggers` enforces "did the mechanism fire?" for *compaction* only; nothing yet does for retrieval/profile/KB engagement, grade provenance, or whether the *grader* saw what it needed. Standing guard for any new arm or metric: before trusting a per-arm number, confirm the signal/label/grader reflects what the config actually did.

Talk-outline correction: the planned "skills / just-in-time KB instructions" change (Change 4) was never built, and its premise is off — the KB instructions block measures **458 tokens** (not ~4k) inside a 1,655-token system prompt; max saving ~2% of a turn, mostly cache-discounted. Recommended reframe: demo progressive disclosure on tool *outputs* (F1), where our numbers actually are.

## Remaining work

**Omar**
- [x] Grade the session probes (the critical tier). 2026-06-15: all 96 reviewed — the 83 high-confidence LLM verdicts by sign-off, the 13 low-confidence ones individually — with **zero overrides**, so the provisional LLM grades are now human-confirmed ground truth (see F13). The non-probe rows (persona / key_point / behavior in `review_other.md`) were not part of this pass and remain LLM-graded; bless them too if those columns need a human label.
- [ ] Audit `data/eval/review_log_v1.md` (the 11 flagged judgment calls first).
- [x] Verify and update the `gemini-3.5-flash` entry in `app/telemetry.py:MODEL_PRICING` against the current price sheet.

**Workshop**
- [ ] Replace placeholder slide numbers with measured ones; pick 2–3 failure traces from the graded probes for the failure→fix→number beats (a `fact_update` failure under summarization is the target demo).
- [ ] Token-vs-turn plot: CSV is ready; add matplotlib (or plot elsewhere) for the PNG.
- [ ] Next.js meter component over the `data-context-stats` SSE part, if the live demo needs it.
- [ ] Skills section: reframe per the 458-token measurement above.

**Beyond the workshop.** Part C itself is done and reported above (findings F14–F24); what remains only *hardens* or *extends* it and is consolidated in "Part C — what remains" above — promotion, finishing the v2 contradiction tier, the `incontext_history_retrieval` long-session test, the conditional subsystem builds, and the deliberately-not-pursued list. The full execution detail (variant wiring, telemetry signals, the subset run matrix, the cost model, and the v2 battery plan) lives in `evals/part_c_plan.md`.

**Product quality track (post-workshop)**
Golden cases as a CI gate (5–10 critical-path cases with deterministic assertions, including the F7 non-empty check) → error-analysis cycles (an expert reads 50–100 traces in a small viewer, records pass/fail plus the first failure reason, and stops when new failure types stop appearing) → failure taxonomy → code assertions for recurring modes → LLM judges built from the hand-grade critiques and validated to >90% true-positive and true-negative rates on held-out human labels before any judge-graded number is reported → nightly battery + weekly trace sampling. Full methodology behind each step: `evals/background.md`.

## Files

- `evals.md` — this file: the what, the data, the results, the queue.
- `evals/contributing.md` — contributor playbook: how to run an experiment and land it (run → upload data to HF → record an `F<N>` finding → merge), with a definition-of-done gate.
- `evals/part_c_plan.md` — Part C execution plan: orchestration (workflow vs direct), the variant catalog with exact wiring + telemetry signals, the subset run matrix, and cost.
- `evals/background.md` — research sources (Hamel, howtoeval, OpenAI macro-evals) and design rationale.
- `evals/graphrag.md` — GraphRAG vs classical RAG experiment (branch `experiment/graphrag-vs-rag`): a scoped, true-GraphRAG head-to-head on the single-turn battery. Revisits the dropped GraphRAG idea with a fair test.
- `evals/compaction.md` — compaction-vs-keep-everything study (branch `experiment/context-compaction`): on the largest lesson, keep-all vs every compaction method vs retrieve-per-question. **Standalone fleet on Gemini 2.5 Flash — not comparable to the 3.5-Flash results above.**
- `evals/slm_compaction.md` — knowledge-compaction on small local models (branch `experiment/slm-compaction`): two experiments on 3 Ollama SLMs (llama3.1:8b, qwen2.5:7b, qwen3:8b) at a 32k window where the lesson overflows. **Axis B** (fit a document: RAG vs stuff vs summarize, F30) and **Axis A** (compact growing history via the real memory presets, F31). The regime where compaction is *forced*; RAG wins Axis B, no universal winner on Axis A, and "keep everything" never wins on an SLM (synthesis in F32).
- `data/eval/README.md` — battery schemas + glossary of terms; `review_log_v1.md` — dataset audit trail.
- `evals/` — the harness code; `app/memory_presets.py`, `app/telemetry.py` — the app-side hooks.
- `runs/b_report/`, `runs/c_report/` — generated tables, token curves, blinded grading workbook (Part B and the Part C screen). Re-grade / follow-up reports: `runs/kbfair_report/` (F23 KB-fair recall), `runs/d_report_profile_memory_active/` (F22 profile rerun), `runs/e2_v2_partial_report/` (v2 partial).
- `runs/ds_v2_t1_report/`, `runs/ds_v2_t2_report/` — **DeepSeek-V4-Flash** v2 matrix (F25/F26: cost + contradiction + long-horizon), graded via the subagent path (`runs/ds_judge/`). Shared on HF as `eval_runs/part_f_deepseek_v2/` (the standard `mv eval_runs/part_*/* runs/` download restores them). Provider added in `app/chat_service.py` (`deepseek` / `openrouter`), pricing in `app/telemetry.py`.
- `runs/deepseek_compaction_stage1_triggerfix_20260715/report/`, `runs/deepseek_structured_only_stage1_20260715/report/` — **DeepSeek long-context compaction study, stage 1** (F35/F36): the 2×2 {`full_history`, summarize@200k} × {raw, stable 10k tool-output cap} factorial, plus the follow-up **prefix-preserving compaction** arm (F37: `exp_c200_cap10k_structured`, `summarization_strategy="structured_prefix"` in `app/chat_service.py` — the summarizer call rides the provider cache instead of LangChain's cache-missing XML re-serialization; −14%/trajectory vs the XML arm). Writeups: `stage1_findings.md` / `structured_findings.md` in each report dir (incl. `audit_20260715.md`, the validity audit); runner `evals.run_compaction_experiment`; graded via the blinded subagent path into `runs/_grading/deepseek_*_20260715/`. Shared on HF as `eval_runs/deepseek_compaction_stage1/` (runs + `_grading/` judge verdicts; restore line in the download snippet above). `battery_sessions_v2.jsonl` is **deprecated for new runs** (python_colab confound — see harness corrections); the runners refuse it, use the repaired `battery_sessions_v2_1.jsonl`.
- `runs/axisa_*_report/` — **SLM compaction** study (F30/F31/F32, branch `experiment/slm-compaction`): Axis B fit-a-document and Axis A compact-history on 3 Ollama SLMs at a 32k window. Shared on HF under `eval_runs/slm_compaction/` (`axis_b/<model>/`, `axis_a/axisa_<model>_<preset>/`) + batteries `eval/compaction/`; the snippet above restores Axis-A to `runs/axisa_*`, Axis-B stays under `eval_runs/slm_compaction/axis_b/` — full restore command in `evals/slm_compaction.md`. Provider `ollama` added in `app/chat_service.py`.