docs(evals): findings F35-F38, battery v2 confound corrections, staleness fixes

- F35: 2x2 factorial on DeepSeek — the cheap win is the stable tool-output
cap (-38%/trajectory, cache intact); summarization adds cost on both sides.
- F36: compaction memory damage tracks lossy-rewrite dose, not live context
size; stated on audit-clean probes after the battery audit.
- F37: prefix-preserving compaction removes the summary-generation cache miss
(0%->94% hit, -14%/trajectory vs stock XML); post-compaction prefix break
is structural, so capped full history stays cheapest. Verified analog of
Codex local compaction (source-level comparison in the run report).
- F38: same-trace repricing at GPT-5.6 Sol rates — the keep-vs-compact
crossover is set by the cached-input price (fh wins iff cached input
< ~$0.55/M at this horizon); adds the fourth boundary to F32.
- Harness corrections: mixed-judge grading confound (one judge per battery);
battery_sessions_v2 systematic recycled-filler confound, repaired as
battery_sessions_v2_1 with the runners refusing v2 for new runs; personas
anti-pattern misfire defects queued.
- Staleness sweep from a four-agent doc audit: rescoped the single-model
claim to the Part B/C screen, corrected key requirements for the DeepSeek
default, fixed the broken report glob in contributing.md, fixed the
F24-blind smoke-test advice in data/eval/README.md, pinned the model in
graphrag.md's reproduce commands, marked superseded plan items.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

data/eval/README.md

@@ -1,6 +1,6 @@
 # Evaluation batteries (v1, frozen 2026-06-11)
 
-The datasets for evaluating the AI tutor (see `evals.md` at the repo root for the overall effort and results). **This README is the single reference for what every file, field, and term means.**
+The datasets for evaluating the AI tutor (see `evals.md` at the repo root for the overall effort and results). **This README is the single reference for what every file, field, and term means.** Files added after the v1 freeze: `battery_sessions_v2.jsonl` (deprecated for new runs — see the v2 note in the probe-type glossary), `battery_sessions_v2_1.jsonl` (its repaired successor), and `compaction/` (the F29/F30 lesson-compaction batteries, documented in `evals/compaction.md` / `evals/slm_compaction.md`).
 
 Built from `data/academy_discussion_eval.jsonl` — real student posts from the academy discussion boards with real staff answers — plus authored content (sessions, personas).
 
@@ -22,7 +22,7 @@ Built from `data/academy_discussion_eval.jsonl` — real student posts from the
 
 | File | Tests | Main metrics it feeds | Graded by |
 |---|---|---|---|
-| `battery_singleturn_v1.jsonl` | One-question chats from real student posts. These check whether the tutor answers correctly and chooses the right kind of response. Memory presets should score about the same here; if they do not, the memory setting may be changing behavior in a bad way. | Key-point coverage, behavior accuracy, whether retrieval found the right source, citation validity, tokens, cost, latency. | Automatic checks plus human grades. |
+| `battery_singleturn_v1.jsonl` | One-question chats from real student posts. These check whether the tutor answers correctly and chooses the right kind of response. Memory presets should score about the same here; if they do not, the memory setting may be changing behavior in a bad way. | Key-point coverage, behavior accuracy, whether retrieval found the right source, citation validity, tokens, cost, latency. | Automatic checks plus human or validated-judge grades (F14). |
 | `battery_sessions_v1.jsonl` | Longer study sessions. Early turns plant facts about the student, middle turns make the conversation large, and later graded turns test whether the tutor still remembers or updates important facts. | Memory probe accuracy, cumulative tokens, whether compaction fired before probes, late-turn retrieval behavior. | Automatic trigger checks plus human or judge grades for nuanced probes. |
 | `battery_personas_v1.jsonl` | Stored student profiles plus fresh questions. Each question is designed so the best answer needs profile details, such as OS, skill level, or goal. | Personalization pass rate. | Mostly automatic regex and anti-pattern checks; a few rows need human or judge review. |
 | `replay_n1_v1.jsonl` | Real discussion threads turned into "what should the next staff reply be?" tests. The tutor receives the real conversation up to just before a staff reply, then writes the next reply. | How close the tutor's reply is to the real staff reply. | Human or validated judge grade. |
@@ -41,7 +41,7 @@ Built from `data/academy_discussion_eval.jsonl` — real student posts from the
 - **requires_notebook**: fully answering needs the Colab notebook contents (cells/outputs/files), which our KB does not contain — verified against lesson markdown with file/line evidence (`notebook_evidence`), not guessed. All 60 included cases are `false`; the one true case in review was excluded. If retrieval misses on a flagged-adjacent case, check `notebook_evidence` before blaming the retriever.
 - **requires_media**: the post referenced a screenshot we don't have. The 2 included cases remain gradeable without it.
 - **planted facts**: personal facts (environment, goal, level, weak topic, preference/constraint) a session's student states in turn 0, which later probes test. The session-level analog of a profile.
-- **Probe types** (`probe_type`): `fact_recall` means the answer must use a fact the student gave many turns ago. `preference_compliance` means the tutor must honor a stated preference, such as OS, tooling, language, or style. `anaphora` means the user asks about "that thing from earlier," so the tutor must resolve the reference. `anaphora_consistency` means the answer must stay consistent with the tutor's own earlier recommendation. `fact_update` means the student changed a fact mid-session; using the old value fails. `behavior_routing` means a support/platform issue appears inside a long chat, and the tutor must still redirect appropriately. Battery v2 (planned, not built) will add `contradiction` (a fact changed early and probed only after it would be evicted), `longhorizon_recall`, and `entity_isolation` (two entities in one session, no fact bleed). `cross_session_recall` was an earlier idea for `sleeptime_consolidation` and is deferred with the multi-thread battery; the v1 types here are frozen. See `evals/part_c_plan.md` → "Battery v2" for the build plan.
+- **Probe types** (`probe_type`): `fact_recall` means the answer must use a fact the student gave many turns ago. `preference_compliance` means the tutor must honor a stated preference, such as OS, tooling, language, or style. `anaphora` means the user asks about "that thing from earlier," so the tutor must resolve the reference. `anaphora_consistency` means the answer must stay consistent with the tutor's own earlier recommendation. `fact_update` means the student changed a fact mid-session; using the old value fails. `behavior_routing` means a support/platform issue appears inside a long chat, and the tutor must still redirect appropriately. Battery v2 (`battery_sessions_v2.jsonl`, built 2026-06-16: 6 sessions, `tier`/`tags` metadata, filler recycled from v1 corpus questions) added `contradiction` (a fact changed early and probed only after it would be evicted — note the eviction is not guaranteed in capped arms; see evals.md F36), `longhorizon_recall`, and `entity_isolation` (two entities in one session, no fact bleed). **Use `battery_sessions_v2_1.jsonl` for new runs**: the 2026-07-15 audit found v2's recycled filler included v1 persona/update/pivot turns whose first-person claims collide with the planted facts (7 turns across 4 sessions; details in evals.md harness corrections); v2.1 replaces those turns with neutral same-course corpus questions, probes and plants unchanged, and the runners refuse v2 without `--allow-deprecated-battery`. `cross_session_recall` was an earlier idea for `sleeptime_consolidation` and is deferred with the multi-thread battery; the v1 types here are frozen. See `evals/part_c_plan.md` → "Battery v2" for the build plan.
 - **expected_facts / check_note**: per probe, the specific facts a correct answer must use, and a one-line pass/fail rule for the grader. Grade probes binary → **memory probe accuracy**, reported overall and per probe type per memory preset.
 - **profile_seed / facts**: the persona's stored profile. Seed it with `app.chat_service.set_student_profile(student_id, profile_seed)` before running that persona's questions; `facts` is the same content structured, for slicing results by fact type.
 - **Self-grading checks** (`checks` / `anti_patterns`): persona questions grade themselves because we authored the ground truth. `{"type": "regex_any", "patterns": [...]}` = case-insensitive regex over the answer; any match passes that check. `{"type": "llm", "instruction": ...}` = a judge call with that instruction (use a strong Anthropic model; binary verdict). `anti_patterns` = case-insensitive regexes whose match FAILS the question (e.g. bash `export` advice for a Windows persona). A question passes when every check passes and no anti-pattern matches.
@@ -96,7 +96,7 @@ Use this file to test memory over a longer conversation. Run every message in `t
 | `tier`, `tags` | Optional v2 metadata for staged runs, for example `tier1_contradiction`, `tier2_longhorizon`, `tier3_entity`. The runner can filter these with `--tags`; v1 sessions usually omit them. |
 
 Probe mix: fact_recall 55 · preference_compliance 27 · anaphora 12 · anaphora_consistency 9 · fact_update 7 (sessions s08, s11, s18, s21, s24, s31) · behavior_routing 3.
-**Before a comparison run**: smoke-run a few sessions and confirm compression actually fired before the probe turns (`context_stats.summary_messages` / `cleared_tool_outputs` > 0); lengthen sessions if not — a memory eval where compaction never triggered measures nothing.
+**Before a comparison run**: smoke-run a few sessions and confirm compression actually fired before the probe turns (`context_stats.summary_messages` > 0, or `evals.check_triggers`; for clearing arms use input-token deltas — `cleared_tool_outputs` is structurally always 0, see evals.md F24); lengthen sessions if not — a memory eval where compaction never triggered measures nothing.
 
 ### `battery_personas_v1.jsonl` — 10 personas × 4 questions
 

evals.md

@@ -18,7 +18,7 @@ Plain-English map:
 
 The system under test is the production tutor (`app/`): the same agent, retrieval tools, prompts, and telemetry used by the app. The tutor can answer from the course/docs corpus through `retrieve_tutor_context`, browse the local knowledge base through `run_kb_command`, and keep per-thread conversation memory.
 
-**The single variable is how the tutor manages context** — 13 arms across two axes (**A** = memory/history, **B** = retrieval/tool-outputs). Everything else is held constant: **model (`gemini-3.5-flash`) for every run — no other model (Claude Haiku, DeepSeek, …) has been evaluated** — plus system prompt, retrieval configuration, selected sources, web tools off, and temperature. Most arms are a `MemoryConfig` preset (`app/memory_presets.py`, part of the agent cache key); a few vary a retrieval knob or a tool toggle instead. In this document, **compaction** means reducing the prompt by summarizing older messages or clearing old tool outputs.
+**The single variable is how the tutor manages context** — 13 arms across two axes in the Part B/C screen (**A** = memory/history, **B** = retrieval/tool-outputs), with later experiments adding more arms (F29's summarization family, F35/F37's `exp_*` factorial + structured-prefix arms; `exp_c400/c800` built, unrun). Within each experiment everything else is held constant — system prompt, retrieval configuration, selected sources, web tools off, temperature, and the model. **The Part B/C screen held the model at `gemini-3.5-flash`; later experiments varied it** (DeepSeek-V4-Flash — now the production default — in F25/F26/F35–F38, Gemini 2.5 Flash in F29, three Ollama SLMs in F30/F31), with per-finding comparability caveats: absolute numbers never cross fleets. Most arms are a `MemoryConfig` preset (`app/memory_presets.py`, part of the agent cache key); a few vary a retrieval knob or a tool toggle instead. In this document, **compaction** means reducing the prompt by summarizing older messages or clearing old tool outputs.
 
 **The arms — every strategy we tested:**
 
@@ -47,9 +47,10 @@ The system under test is the production tutor (`app/`): the same agent, retrieva
 | `battery_sessions_v1` | 32 sessions (113 probes) | memory under compaction | **only 3 sessions ever** (s01/s02/s08), in B and C |
 | `battery_personas_v1` | 10 personas × 4 Qs | long-term profile memory | B (all) |
 | `replay_n1_v1` | 30 "next staff reply" cases | multi-turn behavior on real threads | **built, never run** |
-| `battery_sessions_v2` | 6 sessions, 3 tiers (contradiction / long-horizon / entity) | where `full_history` might lose | **partial** — `prod` all 6, `full_history` 2 Tier-1 only |
+| `battery_sessions_v2` | 6 sessions, 3 tiers (contradiction / long-horizon / entity) | where `full_history` might lose | Gemini partial (`prod` all 6, `full_history` 2 Tier-1), then in full on DeepSeek: the 12-arm v2 matrix (F25/F26) and the ×3-trial stage-1 factorial (F35–F37). **Deprecated 2026-07-15** (filler confound, see harness corrections) |
+| `battery_sessions_v2_1` | v2 with 7 recycled filler turns repaired (probes/plants identical) | same tiers, confound-free | built 2026-07-15; the required v2 successor for new runs |
 
-**What actually ran, by phase:** **Part B** — the 4-arm bake-off (`full_history`, `prod`, `profile_memory`, `aggressive`) over the full single-turn + personas batteries + 3 sessions, ×2 trials (→ F1–F13). **Part C** — the 11-arm screen (the 2 anchors + 9 variants) over a 24-case single-turn subset + 3 sessions, ×1 trial (→ F14–F20, F23). **D** — re-ran `profile_memory` with its store actually engaged (→ F22). **v2** — a partial long-context / contradiction run (→ the open v2 items in "Part C — what remains" below). Everything so far is a coarse screen (1–2 trials); nothing has been run at promotion grade (full batteries × 3 trials).
+**What actually ran, by phase:** **Part B** — the 4-arm bake-off (`full_history`, `prod`, `profile_memory`, `aggressive`) over the full single-turn + personas batteries + 3 sessions, ×2 trials (→ F1–F13). **Part C** — the 11-arm screen (the 2 anchors + 9 variants) over a 24-case single-turn subset + 3 sessions, ×1 trial (→ F14–F20, F23). **D** — re-ran `profile_memory` with its store actually engaged (→ F22). **v2** — a partial long-context / contradiction run (→ the open v2 items in "Part C — what remains" below). Later phases moved off Gemini: the **DeepSeek v2 matrix** (F25/F26), **GraphRAG head-to-head** (F28), the **Gemini-2.5 compaction study** (F29), the **SLM studies** (F30/F31), and the **DeepSeek stage-1 factorial + structured-compaction follow-up** (F35–F38, ×3 trials on the v2 sessions battery). Nothing has been run at promotion grade (full batteries × 3 trials); the earlier phases are 1–2-trial screens.
 
 *Why the B→C order:* Part B's findings drove it — tokens live in tool outputs not history (F1/F3) and compaction can cost more than it saves (F2/F9), so the cheap retrieval/tool-output variants ran first and the history-compaction variants ran to confirm that inversion. That narrative lives in the Results sections below; this section is the complete catalog.
 
@@ -98,16 +99,17 @@ Command roles:
 - `report` builds side-by-side tables and token curves from graded runs.
 - `handgrade_workbook` builds and merges a blinded grading workbook, so the grader does not know which preset produced an answer.
 
-Every turn persists a JSON trace bundle, so grading and reporting can re-run offline without touching the API. Runs are resume-safe: re-run the same command after an interruption and completed units are skipped. Each turn has a 10-minute timeout, and LangSmith is off by default. Running the tutor needs `GEMINI_API_KEY` + `COHERE_API_KEY`; the full 4-preset comparison run below cost **≈$323** at corrected pricing (the pre-2026-06-13 price table under-priced `gemini-3.5-flash` ~4.4×; the often-quoted "$73" is a pre-correction figure — see Harness corrections).
+Every turn persists a JSON trace bundle, so grading and reporting can re-run offline without touching the API. Runs are resume-safe: re-run the same command after an interruption and completed units are skipped. Each turn has a 10-minute timeout, and LangSmith is off by default. Running the tutor needs `COHERE_API_KEY` plus the key for the model under test — `DEEPSEEK_API_KEY` for today's default (with only a Gemini key the app substitutes gemini-2.5-flash outright and the harness flags every bundle); the Part B/C runs used `GEMINI_API_KEY`; the full 4-preset comparison run below cost **≈$323** at corrected pricing (the pre-2026-06-13 price table under-priced `gemini-3.5-flash` ~4.4×; the often-quoted "$73" is a pre-correction figure — see Harness corrections).
 
 **Setup for collaborators.** Code and docs are in git; the datasets and run results contain real student text and live only in the private HF dataset (`towardsai-tutors/ai-tutor-data` — git is force-pushed to the public prod Space on deploys, so student data never enters git). With an `HF_TOKEN` that can read it:
 
 ```bash
 export HF_TOKEN=hf_...    # token with READ access; the inline cmd does NOT load .env (or: uv run huggingface-cli login)
-mkdir -p data/eval runs  # runs/ is gitignored, so a fresh clone has no runs/ dir yet
+mkdir -p data/eval runs/_grading  # runs/ is gitignored, so a fresh clone has no runs/ dir yet
 uv run python -c "from huggingface_hub import snapshot_download as d; d(repo_id='towardsai-tutors/ai-tutor-data', repo_type='dataset', allow_patterns=['eval/**','eval_runs/**'], ignore_patterns=['eval/README.md'], local_dir='.')"
 mv eval/* data/eval/ && mv eval_runs/part_*/* runs/   # restore working paths (part_b, part_c, ...)
 mv eval_runs/slm_compaction/axis_a/* runs/ 2>/dev/null   # SLM Axis-A runs -> runs/axisa_* (F31); Axis-B (F30) stays under eval_runs/slm_compaction/axis_b/, see evals/slm_compaction.md
+mv eval_runs/deepseek_compaction_stage1/deepseek_* runs/ 2>/dev/null && mv eval_runs/deepseek_compaction_stage1/_grading/* runs/_grading/ 2>/dev/null   # DeepSeek stage-1 compaction runs + judge verdicts (F35-F38)
 ```
 
 ## Results — Part B comparison run, 2026-06-12
@@ -189,6 +191,14 @@ Numbered findings; each states what it was tested on. Convention: entries are ne
 
 - **F34 (2026-06-23) — Across a session, standard compaction's damage is *surgical*: it collapses memory while answer quality stays ~100% — but that "quality" is partly an artifact of a history-blind judge, so "looks fine" ≠ "served the student". Only the most aggressive rewrite (`context_reset`) degrades the answers themselves, progressively over the session. (Refines F10/F12 on the holistic axis.)** Holistic graded on every answered turn of `b_se`+`c_se` (683 turns, 15 arms, blind subagent path). **The decoupling:** `prod` probe **58%**(c)/**38%**(b) yet holistic **97/99%**; `kb_off` probe 33% / holistic 100%; `selective_retention` probe 25% / holistic 100%; `profile_memory` probe 38% / holistic 100% — memory and answer-quality are *independent failure axes*. The lone exception is `context_reset` (probe 17%, holistic **69%**, holistic dropping **−24% early→late turn**; `sliding_window` a milder −16% late): broad, progressive quality degradation appears only under the most destructive history rewriting, otherwise quality holds flat (`Δ late−early` ≈ 0 for all standard arms). **But "high holistic" is misleading on exactly the turns that matter** — holistic is graded *blind to the session history* (sessions carry no per-turn gold answer, and the judge sees only question+answer), so on the **memory-failed probe turns** it still rated the (generic, context-ignoring) answer "good" **100% of the time** for every standard arm (`c_se_prod` 5/5, `b_se_prod` 15/15, `kb_off` 8/8, `selective_retention` 9/9; `forgot+looks-bad` = **0**) — only `context_reset` forgot so badly the blind judge caught it (6/10 forgotten-fact turns also looked bad). The failure mode is therefore *confidently generic*: a well-formed answer that silently ignores what the student stated earlier, which only the targeted probe detects. **Lesson:** holistic (context-free quality) and probe (context-aware memory) measure different things and must be read together — "answer quality looks fine" does **not** mean the student was served. This refines F10 (memory dies but general answer quality survives) and F12 (only the *aggressive* rewrites *also* degrade quality). Caveats: `c_se` is 3 sessions × 1 trial (~18 turns/early-late bucket), `b_se` ×2 trials; session holistic graded without a staff reference (the within-arm early/late comparison controls for that); judge reproducibility-validated on probes (F14), not on holistic — the blind-spot above is itself why a held-out human validation of holistic is the next gate.
 
+- **F35 (2026-07-15) — 2×2 factorial on DeepSeek: the cheap win is a *stable* tool-output cap, not compaction — capping cuts cost −38% at no measured quality/memory loss while preserving the cache, and summarization adds cost on BOTH sides of the factorial, so it still doesn't pay for itself even after capping. (Decomposes F25; Axis A × Axis B; DeepSeek-V4-Flash first-party — absolute numbers not cross-comparable with the Gemini fleets.)** *Experiment: **DeepSeek long-context compaction study, stage 1.** The study asks whether summarization-based history compaction ever pays for itself on the production tutor running its default model (DeepSeek-V4-Flash, first-party API) over long tutoring sessions, once prompt caching and tool-output size are accounted for. F25 answered "no, `full_history` stays cheapest through 36 turns" — but its compaction arms confounded two mechanisms, the history policy and the giant raw tool outputs that dominate input tokens (F1), so it couldn't say* where *the money actually goes. Stage 1 is the clean decomposition: a 2×2 factorial crossing the history policy (keep everything vs summarize at a 200k trigger, retaining a 50k recent tail) with tool-output handling (raw vs a stable 10k-token cap applied once when the output enters history), giving four arms — `exp_fh_raw` / `exp_fh_cap10k` / `exp_c200_raw` / `exp_c200_cap10k` (`app/memory_presets.py`) — whose paired contrasts isolate each mechanism's cost/memory effect and their interaction. Stage 2 (trigger-threshold sensitivity: `exp_c400_cap10k` / `exp_c800_cap10k`) is built but not yet run. Method: battery `battery_sessions_v2` (22-/36-turn scripted tutoring sessions that plant student facts early and probe them late), 3 trials, run via the `evals.run_compaction_experiment` paired lockstep runner (all four arms advance turn-by-turn together in randomized within-turn order, each arm/session/trial under a distinct DeepSeek cache `user_id` so no arm warms another's KV cache; immutable run fingerprint; `evals/contributing.md` §2). "triggerfix" in the run name marks this as the corrected rerun after the earlier stage-1 attempt's summarization-trigger bug. Run `runs/deepseek_compaction_stage1_triggerfix_20260715`; full writeup `runs/deepseek_compaction_stage1_triggerfix_20260715/report/stage1_findings.md`; raw on HF `eval_runs/deepseek_compaction_stage1/`.* Four arms {`full_history`, summarize@200k keep-50k} × {raw tool outputs, stable 10k-token cap (40k bytes) applied once when the output enters history} on sessions-v2 Tier-1 contradiction (3×22t) + Tier-2 long-horizon (2×36t), ×3 trials, run turn-level-lockstep with randomized within-turn order and a distinct DeepSeek cache `user_id` per arm/session/trial (no cross-arm KV warming); 414 turns/arm, 0 errors. The earlier stage-1 trigger bug is fixed and evidenced: every compaction fired at ≥200,056 pre-compaction tokens with 92k–198k-token summary inputs, and the `full_history` arms compacted 0 times. Cost (mean $/trajectory): capping alone **$0.189 → $0.117 (−38%, cheaper in 14/15 paired trajectories)** with the cache-hit ratio intact (96.0%→95.9%) — it removes ~1MB of repeated tool payload per trajectory *without* rewriting the prefix, so it composes with caching instead of fighting it (the F1→F24/F25 thread: the tokens live in tool outputs, and cutting them *stably* is nearly free). Summarization is a net cost **add** on both sides: +50% vs raw full-history (14/15) and **+32% vs capped full-history (12/15)**; capping shrinks the compaction penalty (interaction −$0.057) but does not reverse it. Quality holds where it should: holistic 96–98% for all four arms, and the cap costs no memory (probe 87%, identical to raw full-history). Contrast with F19: per-call-view head/tail truncation churned (re-calls, 33% memory); this checkpoint-level insertion-time cap shows no churn at all (3.0 vs 3.1 LLM calls/turn, 87% memory) — though the arms differ in retained size and model too, the stability of what the model re-reads looks like the operative difference. Caveats: probes n=15/arm and only 5 session templates (the writeup's cluster-bootstrap intervals are descriptive, not population estimates); holistic is judge-graded, not human-validated (F34); DeepSeek's ~50× cache discount is load-bearing for the ranking (F25's provider caveat).
+
+- **F36 (2026-07-15) — Where compaction hurts memory, the damage tracks the *number of lossy rewrites*, not the size of the live context — and raw tool outputs double that number, so Axis-B bloat *amplifies* Axis-A damage. "Too many tokens in the answering call" is not the failure mode: `full_history` recalls perfectly from up to 879k-token contexts. (Refines F10/F26's mechanism.)** *Experiment: same DeepSeek stage-1 compaction factorial as F35 (run `runs/deepseek_compaction_stage1_triggerfix_20260715`, battery `battery_sessions_v2`; probe-level evidence in the same `stage1_findings.md` writeup) — this is the memory half of that run's results. The battery's two tiers each target one memory failure mode: Tier 2 long-horizon (2 sessions × 36 turns × 3 trials) plants student facts/constraints early and probes them ~30 turns later, after compaction has had every chance to evict them; Tier 1 contradiction (3 × 22t × 3) plants a fact, updates it mid-session, and probes whether the agent uses the current value. Probes judge-graded via the F14-validated blinded subagent path.* Long-horizon recall (plant facts early in a 36-turn session, probe at the end): the extended 2026-07-15 audit found one of the two Tier-2 sessions (`v2_t2_fullstack_long_persona_36t`) carries recycled "project pivot" filler that outright contradicts the probed PHI constraint, so its cells cannot separate eviction-by-compression from eviction-by-believed-pivot and are excluded (repaired in `battery_sessions_v2_1.jsonl`). On the clean Tier-2 session (`v2_t2_agentic_long_persona_36t`, whose off-persona filler is orthogonal to its probed governance facts; n=3/arm): **0 summarization events → 3/3 and 3/3** (both full-history arms), **~2–3 events → 3/3 and 3/3** (`exp_c200_cap10k` and F37's structured arm), **~5–6 events → 1/3** (`exp_c200_raw`). The dose mechanism survives in coarser form — only the high-dose raw arm measurably loses far-back facts, and low-dose capped compaction is indistinguishable from full history at this n — but the fine "6/6 → 5/6 → 3/6" gradient quoted before the audit mixed in the compromised session and should not be cited. The raw arm compacts twice as often because uncapped tool payloads regrow the history to the 200k trigger faster; each event then compresses ~150k tokens into ~1.7k (~90×), and on the clean session the judge's fail reason is genuine eviction ("explicitly denies knowing the student's constraints and offers only a generic list of common governance options"). Read-time context size points the *other* way: at probe time the compaction arms answered from 95–195k-token contexts and failed, while `full_history` answered from 363–879k and never missed — so the "context rot" here lives in the summarization pipeline (a planted fact that is a tiny fraction of a payload-dominated span must survive every ~90× rewrite), not in the answering model's attention. Separately, the contradiction tier: all 11 contradiction misses across all arms sit in **one session** (`python_colab_to_local_22t`), and the miss mode is *hedging* (recap both environments, ask "which machine are you on?") rather than stale-fact use — but the 2026-07-15 battery audit found that session **confounded**, so treat its cells as unmeasured rather than as a finding: its recycled filler injects a first-person *third* environment ("my iMac", turn 16) *after* the turn-7 update and *before* the turn-21 probe, which makes the hedge a defensible reading of genuinely contradictory history, not a demonstrable memory failure. The other two contradiction sessions are confound-free (audited: zero post-update first-person environment claims) and had **zero misses in any arm** — so this run shows no contradiction gap between arms and leaves F26 standing; whether `full_history` hedges when it truly holds A and A′ needs the repaired session rerun (fix as a new battery version — v1/v2 files are frozen). Caveats: the clean Tier-2 evidence is n=3/arm — read "high dose fails, low dose doesn't", nothing finer; Tier-1 does not guarantee eviction before its probe (all five of `exp_c200_cap10k`'s compaction-dormant trajectories are Tier-1, so in capped arms Tier-1 tests update-adherence, not contradiction-under-eviction); Tier-2's probed constraints are partially guessable from world knowledge (a governance rule for a procurement agent), which inflates every arm's absolute recall equally and biases *against* the dose signal. Aggregate probe accuracy restricted to audit-clean probes (9/arm): full-history arms **9/9 and 9/9**, capped compaction **9/9** (XML) / **9/9** (structured), raw compaction **7/9**.
+
+- **F37 (2026-07-15) — A chunk of "compaction's cost" was an implementation artifact, not physics: LangChain's stock summarizer breaks the provider cache on the summary-generation call itself, and a prefix-preserving compaction request removes that — summarizer cache hit 0%→94%, summarization cost −87%, total −14%/trajectory vs the stock arm, with equal-or-better memory. But the *post-compaction* prefix break is structural, so compaction still costs +14% vs capped full history: F35's ranking stands, with a smaller gap. (Same regime as F35/F36 — DeepSeek-V4-Flash first-party, absolute numbers not cross-comparable with the Gemini fleets.)** *Experiment: **DeepSeek compaction study, stage 1 follow-up — prefix-preserving ("structured") compaction.** The stock `SummarizationMiddleware` serializes the selected messages into a fresh XML prompt to ask for the summary, so even summary *generation* pays 100% cache-miss prices on ~150k-token inputs; `PrefixPreservingCompactionMiddleware` (`app/chat_service.py`, `summarization_strategy="structured_prefix"`) instead sends the *unchanged current request prefix* plus one checkpoint instruction, binding the same model/tools/settings, so the summarizer call rides the same cache as the agent. Single arm `exp_c200_cap10k_structured` — identical to `exp_c200_cap10k` except the strategy — run `runs/deepseek_structured_only_stage1_20260715` on the same `battery_sessions_v2` × 3 trials (414 turns, 0 errors), compared pairwise per session×trial against the four stage-1 arms; graded through the same blinded-subagent pipeline (429 judgments, 0 missing). Writeup `runs/deepseek_structured_only_stage1_20260715/report/structured_findings.md`; raw on HF `eval_runs/deepseek_compaction_stage1/`. A 2026-07-15 source-level check of the open-source Codex CLI confirmed this strategy is a faithful analog of Codex's local compaction — unchanged history + appended instruction, explicitly cache-preserving — while Codex's OpenAI-default *remote v2* path is the provider-native opaque-compaction primitive a client middleware cannot replicate; comparison + adoption ideas in `runs/deepseek_structured_only_stage1_20260715/report/codex_comparison.md`.* Cost: vs the stock XML arm, **−$0.0217/trajectory (−14.0%, cheaper in 11/15 pairs) at identical compaction dose** (1.27 events/trajectory both): the summarizer call's cache-hit ratio goes **0.0% → 94.2%**, so summarization cost falls $0.0276 → $0.0036/trajectory (−87%) — even though the structured arm *bills more* raw input (10.6M vs 9.9M tokens/trajectory; the checkpoint request re-reads the whole prefix at the ~50× cache-read discount). Tokens ≠ dollars, again (F2/F25). vs capped full history it remains **+13.7% (cheaper in only 4/15)**: what's left is the structural boundary — installing the summary necessarily rewrites the *next* agent call's prefix, which no client-side middleware can avoid (that would take a provider-native compaction/continuation primitive) — plus the extra agent work (3.5 vs 3.1 LLM calls/turn). Memory: equal-best of the compaction arms — on audit-clean probes (excluding the two battery-compromised sessions, see F36) it scores **9/9**, tied with both full-history arms and capped-XML compaction, while raw compaction scores 7/9; the pre-audit "6/6 vs 5/6" long-horizon edge over the XML arm rode entirely on the compromised fullstack session and should not be cited — whether the intact-prefix summary *also* preserves facts better than the XML rewrite is unmeasured at this n. Holistic flat at 96%. Its single overall probe miss is a contradiction hedge in `python_colab_to_local_22t` — the confounded session — a battery artifact, not an arm signal. Trigger gate: compaction fired in 12/15 trajectories (19 events, min pre-compaction observation 200,053 tokens; the 3 dormant trajectories all Tier-1, and 100% of Tier-2 probes ran under compaction). Caveats: the memory edge is 1–2 probes at n=6 long-horizon / n=15 total — directional; the arm ran solo, not turn-level-interleaved with the stage-1 arms (token/cost metrics are run-time-independent, but treat its latency rows as confounded, F27); holistic judge not human-validated (F34); DeepSeek's ~50× cache discount is load-bearing (F25's provider caveat).
+
+- **F38 (2026-07-15) — The keep-everything-vs-compact crossover is set by one number: the cached-input price. Repricing our stage-1 traces call-by-call shows "full history wins" is a fact about DeepSeek's pricing, not a law: at GPT-5.6 Sol's public rates the *same recorded workload* already flips to favor compaction at 36 turns — quantitatively consistent with why OpenAI's Codex auto-compacts at ~272k while our tutor, on DeepSeek, should not compact at all. (Analysis-only finding — no new run; adds the missing fourth boundary to F32's list.)** *Experiment: same-trace repricing over the stage-1 arms (runs `runs/deepseek_compaction_stage1_triggerfix_20260715` + `runs/deepseek_structured_only_stage1_20260715`, F35/F37): every recorded model call's (fresh, cached, output) token triple re-costed at GPT-5.6 Sol's public API rates ($5 / $0.50 / $30 per M — a 10× cache discount vs DeepSeek's ~50×), with and without the API's >272k long-context cliff (2× input / 1.5× output on the whole request; per OpenAI's Tibo, the cliff is NOT charged on Codex subscriptions — the stated real driver is accumulated cache-read cost scaling with the live window re-read on every tool call).* Result: `exp_fh_cap10k` vs `exp_c200_cap10k_structured` goes from **$0.117 vs $0.134/trajectory (full history −13%) at DeepSeek prices** to **$9.85 vs $9.94 (≈tied) at GPT rates** — and the tie hides a horizon split: full history still wins the 22-turn tier ($6.29 vs $6.76) but **already loses the 36-turn tier ($15.18 vs $14.72), no pricing cliff involved** (directional: 3/6 tier-2 pairs, mean −$0.46; the API cliff widens it to $12.56 vs $9.94 overall, 191/1,247 full-history calls exceeding 272k). Cache reads are **60%** of full history's repriced spend (vs ~28% at DeepSeek prices) — exactly OpenAI's named mechanism, visible in our own traces. Closed form at this 22/36-turn horizon: full history wins iff cached input costs < **~$0.55/M** — DeepSeek charges $0.0028 (200× under the line), GPT-5.6 Sol $0.50 (right at it), so the crossover turn falls from "beyond everything we tested" to "inside ordinary sessions" purely by the price constant. Reading: F9/F15/F25/F35's "keep everything wins" carries an implicit precondition F32's list omitted — **a deep-enough cache discount** — the fourth boundary alongside F32's three (static doc, small window, no caching). It rationalizes Codex's design (compact once cache-read accumulation crosses the compaction tax; not earlier than quality forces, since compaction damage is dose-responsive — F36 — matching Codex's own repeated-compaction accuracy warning; and don't buy bigger windows for quality — flat above 272k per OpenAI, consistent with F6/F26/F35 where 880k-token full-history contexts stayed perfectly accurate) but does **not** derive the specific 272k, which is OpenAI-specific (GPT-5.5-lineage standard-context serving boundary; the API cliff sits exactly there; a Codex 372k trial was reverted for usage cost). Caveats: same-trajectory repricing assumes GPT-5.6 would reproduce DeepSeek's call pattern and ~96% cache-hit rate (imperfect caching hurts full history *more* — F25's routing note — so the real crossover is likely earlier, not later); excludes cache-write billing on both sides; subscription "usage" is OpenAI-internal accounting, not API dollars; trigger sensitivity is unrun (`exp_c400_cap10k`/`exp_c800_cap10k` built, pending), so the optimal-trigger curve is unknown even on DeepSeek.
+
 **Screen winners → promotion candidates:** `incontext_history_retrieval` (needs the long-session test), `retrieval_budget_30k` and `clear_retrieval_kb` (cheap Axis-B wins; clear_retrieval_kb is cheaper than prod with similar memory + better recall/key-points), anchored by `full_history`. **Drop:** `context_reset`, `observation_truncation`, `selective_retention`.
 
 **Part C — what remains (beyond the workshop).** The screen answered the core question; everything below *hardens* or *extends* it and is **gated on a spend or data decision from Omar** — none is required for the workshop.
@@ -200,7 +210,7 @@ Numbered findings; each states what it was tested on. Convention: entries are ne
 
 **Deliberately not pursued.** *Stretch (heavier, likely later):* sub-agent isolation, reframed as a tool-output-token play — keep noisy `run_kb_command` output out of the main context (ties to F1); temporal-graph memory is already the conditional build in item 4 above. *Dropped as low-information given the findings:* the skills / lazy-prompt-loading family (only a ~458-token win — see the talk-outline correction below), procedural memory, GraphRAG (now **empirically tested** rather than dropped on faith — **F28**: ties classical RAG on grounding, costs +44% $/turn, no payoff on single-hop Q&A), and multi-agent / parallel-research agents. Context for the `kb_off` arm (F17/F23): the KB is the dominant grounding tool — ≈89% of Part B turns, ~7.7 KB vs ~0.9 retrieval calls/turn over 1,088 turns — so disabling it forces the 100k-token retrieval fallback, which is why it raised cost/latency.
 
-**V2 execution note (2026-06-16).** A private/gitignored 6-session `battery_sessions_v2.jsonl` was built and partially run. The full v2 screen is too slow under the lower-tier Gemini key: `full_history` at concurrency 3 hit the 3M input-tokens/minute quota, and concurrency 1 works but makes the full 4-arm screen a multi-hour job. Current local artifacts (`runs/e2_v2_partial_report/report.md`) are directional, not final: `prod` completed all 6 sessions with 0 errors, all probes under compaction, **14% probe accuracy (1/7)** and contradiction **0/3**; `full_history` completed 2 Tier-1 contradiction sessions with 0 errors, no compaction, contradiction **2/2**. Priority is now **Tier 1 only**: finish the missing `full_history` contradiction session, then run Tier-1 `profile_memory` and optionally Tier-1 `incontext_history_retrieval`.
+**V2 execution note (2026-06-16).** A private/gitignored 6-session `battery_sessions_v2.jsonl` was built and partially run. The full v2 screen is too slow under the lower-tier Gemini key: `full_history` at concurrency 3 hit the 3M input-tokens/minute quota, and concurrency 1 works but makes the full 4-arm screen a multi-hour job. Current local artifacts (`runs/e2_v2_partial_report/report.md`) are directional, not final: `prod` completed all 6 sessions with 0 errors, all probes under compaction, **14% probe accuracy (1/7)** and contradiction **0/3**; `full_history` completed 2 Tier-1 contradiction sessions with 0 errors, no compaction, contradiction **2/2**. Priority is now **Tier 1 only**: finish the missing `full_history` contradiction session, then run Tier-1 `profile_memory` and optionally Tier-1 `incontext_history_retrieval`. **(2026-07-15: superseded — the stage-1 factorial ran the full battery ×3 trials on DeepSeek (F35–F37), and the contradiction cells here sit on the session the audit later found confounded; new runs must use `battery_sessions_v2_1.jsonl`.)**
 
 Harness corrections (bugs in our measurement, not findings): the overnight 06-12 stall was machine sleep hanging API streams (all four pipelines stopped the same minute; fixed with the per-turn timeout); battery lesson-URLs carried a `/discussions/` suffix that silently zeroed recall@lesson until normalized (caught by the A3 smoke, re-graded from bundles without re-running).
 
@@ -208,11 +218,13 @@ Harness corrections (bugs in our measurement, not findings): the overnight 06-12
 - **Stable `sheet_row_id` (2026-06-15).** `evals.grade` built `sheet_row_id` with builtin `hash()`, which is salted per process, so regenerating a `handgrade_sheet.csv` produced ids that no longer matched the frozen workbook keymap — silently emptying `handgrade_workbook merge`. Switched to a `hashlib.md5` hash; the Part B re-merge was rebuilt via the deterministic `run_id` to recover the human grades.
 - **`cleared_tool_outputs` is blind to library clearing (2026-06-19).** `ContextEditingMiddleware` edits a per-call `deepcopy` and never persists the placeholder to the checkpoint that `context_window_stats` reads, so the signal is 0 for all presets whether or not `ClearToolUsesEdit` fired (full detail + token evidence in **F24**). Detect clearing via input-token deltas; the custom Part C view-mechanisms avoid this by reporting through the `app.telemetry` turn-signal registry.
 
-- **Faithfulness parked; bundle tool-output cap raised 6k→40k (2026-06-23).** The new **faithfulness** rubric (groundedness vs retrieved evidence, F33) is *not reportable on existing runs*: `evals/run_battery.py` capped each saved tool output at `TOOL_OUTPUT_MAX_CHARS = 6_000` while the KB shell returns up to **40k** (`app/kb_shell.py` `DEFAULT_MAX_OUTPUT_CHARS`), so on KB-heavy arms the judge saw a fraction of what the agent actually grounded on and scored *capture-completeness, not grounding* — faithfulness tracked the KB:retrieval ratio almost monotonically (`b_st_prod` kb:ret 10.3 → **4%**; `c_st_prod` 7.7 → 25%; `kb_off`/`graphrag` kb:ret 0 → 50–76%), the same blind-signal class as **F23/F24**. Raised the cap to **40k** (captures what the agent saw; `output_chars` still records the true length so any overflow stays visible) so *future* re-records make faithfulness measurable; existing bundles cannot be backfilled (content past 6k is gone — needs a re-run). Until then `evals/grade.py` emits faithfulness **only when evidence was captured in full** (`evidence_is_complete`, self-re-enabling after the cap raise), and `evals/report.py` + `evals/grading_merge.py` drop the column (`DROP_ITEM_TYPES`). Holistic (F33/F34) is unaffected — it grades the answer, not the evidence.
+- **Faithfulness parked; bundle tool-output cap raised 6k→40k (2026-06-23).** The new **faithfulness** rubric (groundedness vs retrieved evidence, F33) is *not reportable on existing runs*: `evals/run_battery.py` capped each saved tool output at `TOOL_OUTPUT_MAX_CHARS = 6_000` while the KB shell returns up to **40k** (`app/kb_shell.py` `DEFAULT_MAX_OUTPUT_CHARS`), so on KB-heavy arms the judge saw a fraction of what the agent actually grounded on and scored *capture-completeness, not grounding* — faithfulness tracked the KB:retrieval ratio almost monotonically (`b_st_prod` kb:ret 10.3 → **4%**; `c_st_prod` 7.7 → 25%; `kb_off`/`graphrag` kb:ret 0 → 50–76%), the same blind-signal class as **F23/F24**. Raised the cap to **40k** (captures what the agent saw; `output_chars` still records the true length so any overflow stays visible) so *future* re-records make faithfulness measurable; existing bundles cannot be backfilled (content past 6k is gone — needs a re-run). Until then `evals/grade.py` emits faithfulness **only when evidence was captured in full** (`evidence_is_complete`, self-re-enabling after the cap raise), and the column is dropped downstream (`grading_merge.DROP_ITEM_TYPES`; `report.py` omits it directly). Holistic (F33/F34) is unaffected — it grades the answer, not the evidence.
 - **Report mislabeled judge grades as human (2026-06-19, fixed).** `evals.report` hardcoded "(human grade)" on the behavior / key-point / probe-accuracy rows, but Part C runs were filled by the LLM judge (`judge_filled.csv`), not a person — so `runs/c_report/report.md` asserted a human label over judge numbers. Fix: `evals.grade` now records a `grade_source` per merged grade (the judge prefixes its note `[judge…]`; humans do not), and `evals.report` tags each column `[judge]`/`[human]`/`[mixed]` and renames the rows to "(graded)". Numbers are unchanged (and judge≈human per F14); only the provenance label is now honest. Same label-vs-measurement class as F23/F24. All Part C arms were re-graded and `c_report` regenerated.
 - **`input tok/turn` is billed-across-calls, not context size (2026-06-19, clarified).** Telemetry sums `usage_metadata` over every internal model call in a turn, so a high-loop arm (`observation_truncation`, 22.8 calls) bills lots of input from small contexts while a big-payload arm (`kb_off`) bills similar totals from few large ones. Correct for *cost*, but it is not "how big is the context window." The report now shows both: **input tok/turn (billed, all calls)** and a new **context tokens/turn (window size)** row (from `context_tokens_approx`, always computed but previously unsurfaced). On single-turn the two even rank arms differently (e.g. `kb_off` has the *largest* window at 73k from its 100k retrieval payloads, but mid-pack billed input). Read F1/F6's "~200k input" as billed volume, not window size.
 - **Latency rows are confounded by time-of-run (caveat, not yet fixed).** `run_part_c_screen.sh` runs arms sequentially, and TTFT/total-ms depend on API load at that moment, so cross-arm latency deltas (e.g. F19's 430s p95, F23's 19s-vs-38s) partly reflect *when* each arm ran. Token/cost/recall are run-time-independent and unaffected. Any latency claim that drives a decision should come from arms run interleaved (or repeated trials), not a single sequential screen. **`total − ttft` does *not* remove this** — it is only the answer-streaming tail (~6s, preset-insensitive); the agentic latency incl. the summarization tax is all in `ttft`, so subtracting it deletes the signal. Report the unconfounded work metrics (LLM calls/turn, tokens/turn) as the headline and treat seconds as supporting. See **F27**.
 - **Latency de-confounding tooling (2026-06-22).** Acting on F27: (1) `runs/run_part_c_screen_interleaved.sh` runs the screen question-outer / arm-inner (writes `runs/ci_*`, not `runs/c_*`) so every question's arms share one API-load window — same total runs as the sequential screen, so interleaving is free. (2) `context_stats` now also emits `time_to_first_token_ms` (first reasoning/tool/text token, always ≤ `ttft_ms`), separating raw model responsiveness from the tool-call loop that `ttft_ms` includes; it flows through `evals/grade.py` to the report and shows `—` on runs recorded before it existed. (3) `evals/report.py` now leads with the run-time-independent work metrics (LLM/tool calls/turn, tokens/turn) and tags every latency row `[confounded]`, with a runtime note. All `runs/*report*` were regenerated (layout/labels only; numbers unchanged). The scripts live in gitignored `runs/` like the rest of the run tooling.
+- **Mixed-judge grading confound (2026-07-15, fixed).** While grading the DeepSeek stage-1 compaction factorial (F35/F36, `runs/deepseek_compaction_stage1_triggerfix_20260715`), the first quality merge silently mixed two judges: an interrupted Codex (`gpt-5.6-sol`) subscription pass (`evals.run_subscription_grading`) had graded chunks 000–011 of the grading plan (`runs/_grading/deepseek_compaction_stage1_full_20260715`) — almost exactly the `exp_c200_cap10k` rows — before the blinded subagent workflow (`evals/grade_workflow.js`) graded the rest. The GPT judge is one-directionally harsher on holistic: on the 462 double-graded rows, 67% agreement, with GPT failing 151 rows the subagent judge passed and zero the reverse (fail rates 35% vs 3%; its extra fails are mostly technical-accuracy critiques the blind rubric can't adjudicate without evidence). That made `exp_c200_cap10k`'s holistic read 63% vs 96–98% for the other arms — a grader artifact, not an arm effect (the same rows re-graded by the subagent judge score 98%; the tell was the straddling chunk, where the *same arm* scored 76% under GPT and 97% under the subagent judge). Fix: chunks 000–011 re-graded so all 44 chunks share the F14-validated subagent judge; the Codex verdicts are preserved for comparison (`runs/_grading/deepseek_compaction_stage1_full_20260715/codex_partial/`). Lessons: **one judge per battery** — grade provenance is part of the measurement (same class as the judge-as-human label bug above); and two frontier judges disagreeing this much, one-directionally, on holistic is further evidence that holistic needs the held-out human validation F34 called for.
+- **`battery_sessions_v2` filler recycling is systematically confounded (2026-07-15, two audit passes; repaired as `battery_sessions_v2_1.jsonl`).** v2's filler was recycled verbatim from v1 turns, and the recycling blindly included v1 *persona/fact-update/project-pivot* turns — each a first-person state claim — dropped into a different persona: 7 such turns across 4 of the 6 sessions. Two sessions are unusable as authored: `v2_t1_python_colab_to_local_22t` (a "my iMac" turn injects a third environment between the update and the open-ended contradiction probe — every stage-1 contradiction miss in every arm sits here) and `v2_t2_fullstack_long_persona_36t` (two "project pivot" turns relabel the student's project to hosted architectures that contradict the probed no-PHI constraint, so its long-horizon cells measure the injected contradiction, not memory); two more are muddied (`v2_t2_agentic_long_persona_36t`, `v2_t3_fullstack_two_projects_20t` — injected claims orthogonal to the probed facts). F36/F37 were corrected to audit-clean probe subsets. The same audit cleared everything else: `battery_sessions_v1` (all 32 sessions — no confounds; the update/pivot turns there are the *intended* probed facts), `battery_singleturn_v1`, and `replay_n1_v1` all clean and matching their README counts. Separate defect: 5 of `battery_personas_v1`'s authored anti-pattern checks are negation-insensitive substrings that false-fail correct answers (e.g. anti `import openai` matches the official Node SDK's `import OpenAI from 'openai'`; anti `install Python` matches "you don't need to install Python") — an unquantified downward bias on Part B's persona auto-check rows (F8/F11); repair queued. Fixes shipped: `battery_sessions_v2_1.jsonl` replaces the 7 recycled turns with neutral same-course corpus questions (probes/plants byte-identical), and the runners now **refuse `battery_sessions_v2.jsonl`** for new runs (`--allow-deprecated-battery` reproduces historical runs only). Standing lesson: recycled real-student filler must be scanned for first-person state claims against the planted facts before a battery freezes.
 - **Pattern worth naming — most of our measurement bugs are telemetry/label blind-spots, not wrong math.** Six now share one shape: a number or label asserted a property the instrumentation could not observe for that config — **F21** (mechanism dormant: `profile_memory` never engaged on sessions), **F23** (recall blind to KB browsing), **F24** (clearing invisible to the checkpoint signal), the judge-as-human label above, **faithfulness graded on truncated evidence** (the 6k cap above — the judge scored what it could *see*, not what the agent *saw*), and **F34's holistic-on-session blind spot** (the quality judge sees no conversation history, so it credits answers that silently ignored the student's earlier-stated facts — "looks good" on 100% of memory-failed turns). `check_triggers` enforces "did the mechanism fire?" for *compaction* only; nothing yet does for retrieval/profile/KB engagement, grade provenance, or whether the *grader* saw what it needed. Standing guard for any new arm or metric: before trusting a per-arm number, confirm the signal/label/grader reflects what the config actually did.
 
 Talk-outline correction: the planned "skills / just-in-time KB instructions" change (Change 4) was never built, and its premise is off — the KB instructions block measures **458 tokens** (not ~4k) inside a 1,655-token system prompt; max saving ~2% of a turn, mostly cache-discounted. Recommended reframe: demo progressive disclosure on tool *outputs* (F1), where our numbers actually are.
@@ -248,4 +260,5 @@ Golden cases as a CI gate (5–10 critical-path cases with deterministic asserti
 - `evals/` — the harness code; `app/memory_presets.py`, `app/telemetry.py` — the app-side hooks.
 - `runs/b_report/`, `runs/c_report/` — generated tables, token curves, blinded grading workbook (Part B and the Part C screen). Re-grade / follow-up reports: `runs/kbfair_report/` (F23 KB-fair recall), `runs/d_report_profile_memory_active/` (F22 profile rerun), `runs/e2_v2_partial_report/` (v2 partial).
 - `runs/ds_v2_t1_report/`, `runs/ds_v2_t2_report/` — **DeepSeek-V4-Flash** v2 matrix (F25/F26: cost + contradiction + long-horizon), graded via the subagent path (`runs/ds_judge/`). Shared on HF as `eval_runs/part_f_deepseek_v2/` (the standard `mv eval_runs/part_*/* runs/` download restores them). Provider added in `app/chat_service.py` (`deepseek` / `openrouter`), pricing in `app/telemetry.py`.
-- `runs/axisa_*_report/`, `data/compaction_slm_*/` — **SLM compaction** study (F30/F31/F32, branch `experiment/slm-compaction`): Axis B fit-a-document and Axis A compact-history on 3 Ollama SLMs at a 32k window. Shared on HF under `eval_runs/slm_compaction/` (`axis_b/<model>/`, `axis_a/axisa_<model>_<preset>/`) + batteries `eval/compaction/`; the snippet above restores Axis-A to `runs/axisa_*`, Axis-B stays under `eval_runs/slm_compaction/axis_b/` — full restore command in `evals/slm_compaction.md`. Provider `ollama` added in `app/chat_service.py`.
+- `runs/deepseek_compaction_stage1_triggerfix_20260715/report/`, `runs/deepseek_structured_only_stage1_20260715/report/` — **DeepSeek long-context compaction study, stage 1** (F35/F36): the 2×2 {`full_history`, summarize@200k} × {raw, stable 10k tool-output cap} factorial, plus the follow-up **prefix-preserving compaction** arm (F37: `exp_c200_cap10k_structured`, `summarization_strategy="structured_prefix"` in `app/chat_service.py` — the summarizer call rides the provider cache instead of LangChain's cache-missing XML re-serialization; −14%/trajectory vs the XML arm). Writeups: `stage1_findings.md` / `structured_findings.md` in each report dir (incl. `audit_20260715.md`, the validity audit); runner `evals.run_compaction_experiment`; graded via the blinded subagent path into `runs/_grading/deepseek_*_20260715/`. Shared on HF as `eval_runs/deepseek_compaction_stage1/` (runs + `_grading/` judge verdicts; restore line in the download snippet above). `battery_sessions_v2.jsonl` is **deprecated for new runs** (python_colab confound — see harness corrections); the runners refuse it, use the repaired `battery_sessions_v2_1.jsonl`.
+- `runs/axisa_*_report/` — **SLM compaction** study (F30/F31/F32, branch `experiment/slm-compaction`): Axis B fit-a-document and Axis A compact-history on 3 Ollama SLMs at a 32k window. Shared on HF under `eval_runs/slm_compaction/` (`axis_b/<model>/`, `axis_a/axisa_<model>_<preset>/`) + batteries `eval/compaction/`; the snippet above restores Axis-A to `runs/axisa_*`, Axis-B stays under `eval_runs/slm_compaction/axis_b/` — full restore command in `evals/slm_compaction.md`. Provider `ollama` added in `app/chat_service.py`.

evals/background.md

@@ -176,6 +176,8 @@ When a variant regresses, don't stop at the number: diff the failure taxonomies
 
 ## 8. Implementation notes (grounded in our code)
 
+*Status (2026-07-15): everything in this section has since been built — memory presets (`app/memory_presets.py`), the runner/bundles/telemetry (`evals/run_battery.py`, `context_stats`), and the trigger gate (`evals/check_triggers.py`). See evals.md "How it runs". Line references below are historical.*
+
 1. **Runner**: a Python script calling `stream_chat(ChatRequest(...))` directly (no HTTP needed) — `app/chat_service.py` is the single entry point. Multi-turn sessions: reuse one `thread_id` across turns (InMemorySaver is in-process, so the runner keeps state naturally). Persist one **trace bundle** JSONL per run: case_id, variant, trial, every ChatEvent, per-call token usage, timings, final answer.
 2. **Parameterize the middlewares**: `build_agent` hard-codes ContextEditing/Summarization params (`app/chat_service.py:832–851`). Add a memory-config parameter (env var or ChatRequest field) selecting variant presets, and include it in the agent cache key. This is the one refactor the harness requires.
 3. **Token/latency capture**: LangChain returns `usage_metadata` per model call; also already in LangSmith runs (tagged with model/thread; add `memory_variant` to the metadata dict at `app/chat_service.py:979–992`). Capture both — bundles for offline analysis, LangSmith for trace browsing during error analysis.
@@ -203,6 +205,8 @@ When a variant regresses, don't stop at the number: diff the failure taxonomies
 
 ## 10. The two-week workshop cut (what to actually do before the talk)
 
+*Status (2026-07-15): shipped — see evals.md Parts B/C and the findings log.*
+
 The full plan above is research-grade; the talk needs **demo-grade numbers on a meter plus one honest bake-off table**. The cut:
 
 **Keep (the talk literally requires these):**

evals/compaction.md

@@ -1,6 +1,6 @@
 # Compaction study: compaction methods vs keeping everything in context
 
-A **standalone** experiment (branch `experiment/context-compaction`) answering the
+A **standalone** experiment (developed on branch `experiment/context-compaction`, since merged) answering the
 workshop's core question: when a long context is established and then queried,
 should you **keep it all** (and lean on caching), **compact it** (summarize /
 trim / sliding window / selective / delta / hierarchical), or **not keep it and
@@ -119,13 +119,14 @@ per-turn summarization calls.
 context window, where the 37.5k-token lesson does not fit and "shove it all"
 becomes physically impossible — so compaction stops being optional. That arm is
 a **separate experiment** (different model; its own fleet, never compared to the
-Gemini numbers above) and is developed in the SLM PR, not here.
+Gemini numbers above); it lives in `evals/slm_compaction.md` (since merged).
 
 ## Run it
 
 ```bash
 uv run --env-file .env -m evals.compaction_study build --questions 15
-bash evals/run_compaction_study.sh        # Family A presets, 2.5 Flash, no tools
+PRESETS="full_history prod aggressive sliding_window prompt_compression selective_retention context_reset incontext_history_retrieval delta_summarization hierarchical_summarization" \
+  bash evals/run_compaction_study.sh      # the script's default PRESETS omits the last two
 uv run --env-file .env -m evals.knowledge_compaction --questions 15 \
     --strategies rag graphrag --out data/compaction      # Family B
 uv run --env-file .env -m evals.compaction_study report --runs 'runs/compaction_*'

evals/contributing.md

@@ -18,11 +18,70 @@ New probe types need a frozen battery file `data/eval/<battery>.jsonl` — **nev
 ```bash
 uv run --env-file .env -m evals.run_battery --battery <file> --preset <arm> --out runs/<exp>_<arm>
 uv run -m evals.grade  --run  runs/<exp>_<arm>
-uv run -m evals.report --runs 'runs/<exp>_*' --out runs/<exp>_report
+uv run -m evals.report --runs runs/<exp>_*   --out runs/<exp>_report   # unquoted: report.py takes literal paths, the shell expands the glob
 ```
 
 Keep the **model and regime explicit** (which model is *under test*, context-window size, tools on/off). Runs cost real API money; every turn saves a JSON bundle, so grading/reporting re-run offline for free.
 
+For the DeepSeek long-context compaction study, use the paired runner rather
+than launching each preset sequentially. It advances all arms in turn-level
+lockstep, randomizes their within-turn order, and assigns a distinct DeepSeek
+`user_id` to every arm/session/trial so one arm cannot warm another's KV cache.
+The staging controls (`--first-pair-id`, `--max-pairs-this-invocation`) are
+operational and excluded from the immutable fingerprint. To gate the final run
+on one representative completed pair, first run the final configuration with:
+
+```bash
+uv run --env-file .env -m evals.run_compaction_experiment \
+  --battery data/eval/battery_sessions_v2_1.jsonl \
+  --tags tier1_contradiction tier2_longhorizon --trials 3 \
+  --arm-concurrency 4 --pair-concurrency 2 \
+  --first-pair-id v2_t1_python_colab_to_local_22t \
+  --max-pairs-this-invocation 1 \
+  --out runs/deepseek_compaction_stage1
+```
+
+After inspecting that pair, repeat without `--first-pair-id` and
+`--max-pairs-this-invocation`; the same manifest resumes the remaining pairs.
+
+```bash
+uv run --env-file .env -m evals.run_compaction_experiment \
+  --battery data/eval/battery_sessions_v2_1.jsonl \
+  --tags tier1_contradiction tier2_longhorizon \
+  --trials 3 --arm-concurrency 4 --pair-concurrency 2 \
+  --out runs/deepseek_compaction_stage1
+
+for arm in exp_fh_raw exp_fh_cap10k exp_c200_raw exp_c200_cap10k; do
+  uv run -m evals.grade --run "runs/deepseek_compaction_stage1/$arm"
+done
+
+uv run -m evals.check_triggers \
+  --runs runs/deepseek_compaction_stage1/exp_c200_raw \
+         runs/deepseek_compaction_stage1/exp_c200_cap10k \
+  --min-compactions 1 --min-summary-input 4001 \
+  --expected-trigger-tokens 200000 --first-pre-tokens-min 200000
+
+uv run -m evals.report \
+  --runs runs/deepseek_compaction_stage1/exp_fh_raw \
+         runs/deepseek_compaction_stage1/exp_fh_cap10k \
+         runs/deepseek_compaction_stage1/exp_c200_raw \
+         runs/deepseek_compaction_stage1/exp_c200_cap10k \
+  --out runs/deepseek_compaction_stage1/report
+```
+
+The paired runner retries a transient aborted turn at most twice after the
+initial attempt. The summarizer independently retries transient failures or an
+empty response at most twice. Failed stream attempts are recorded in each
+bundle with `failed_attempt_usage_unavailable=true`, because a dropped stream
+does not deliver DeepSeek's terminal usage chunk. Use pair concurrency 2 for
+the full run: all four arms remain concurrent within a turn, while peak arm
+pressure falls from 12 to 8 compared with pair concurrency 3.
+
+Every run directory now contains an immutable manifest/fingerprint. A resume
+fails if the battery, resolved preset, source tree, dependency lock, pricing
+snapshot, model, or runner arguments changed; use a new output directory rather
+than mixing stale and current bundles.
+
 ## 3. Upload the data to HF — the step that's easy to forget
 
 Batteries → `eval/`, run results → `eval_runs/<experiment>/` on the private dataset. Do this **as part of the experiment, before or with the merge** — not "later."
@@ -40,7 +99,7 @@ api.upload_folder(folder_path="runs", path_in_repo="eval_runs/<experiment>",
 PY
 ```
 
-Then **update the collaborator download snippet in `evals.md`** so a fresh clone restores your runs too — it currently restores `eval_runs/part_*/*`; add a line for your `eval_runs/<experiment>/`. (Existing examples on HF: `eval_runs/graphrag/`, `eval_runs/slm_compaction/`, `eval_runs/part_*`.)
+Then **update the collaborator download snippet in `evals.md`** so a fresh clone restores your runs too — it currently restores `eval_runs/part_*/*`, `eval_runs/slm_compaction/axis_a/*`, and `eval_runs/deepseek_compaction_stage1/*`; add a line for your `eval_runs/<experiment>/`. (Existing examples on HF: `eval_runs/graphrag/`, `eval_runs/slm_compaction/`, `eval_runs/part_*`, `eval_runs/part_f_deepseek_v2/`, `eval_runs/deepseek_compaction_stage1/`.)
 
 ## 4. Write the experiment writeup
 

evals/graphrag.md

@@ -6,7 +6,7 @@ get a defensible answer to "does GraphRAG beat our classical RAG for the tutor?"
 
 GraphRAG was previously **dropped** from the Part C plan ("low information given
 the findings", `evals.md`). This revisits it deliberately, with a fair head-to-head
-rather than on faith. It lives entirely on branch `experiment/graphrag-vs-rag`
+rather than on faith. It was developed on branch `experiment/graphrag-vs-rag` (since merged — the retriever, `graphrag` extra, and runner flags are on `main`)
 and is fully opt-in: the default retriever and all production behavior are
 unchanged.
 
@@ -116,14 +116,18 @@ changing the baseline). Both arms, same 41 cases, Gemini 3.5 Flash, scoped to
 the source:
 
 ```bash
+# fullstack_case_ids.txt is a gitignored local artifact: it is the 41 case_ids from
+# battery_singleturn_v1.jsonl whose source_key == full_stack_ai_engineering.
 IDS=$(cat data/graphrag/fullstack_case_ids.txt)
 # classical RAG arm
 uv run -m evals.run_battery --battery data/eval/battery_singleturn_v1.jsonl \
-    --preset prod --retriever classical --ids $IDS --scope-sources \
+    --preset prod --retriever classical --ids $IDS --scope-sources --disable-kb \
+    --model google-genai:gemini-3.5-flash \
     --out runs/grag_classical
 # GraphRAG arm
 uv run -m evals.run_battery --battery data/eval/battery_singleturn_v1.jsonl \
-    --preset prod --retriever graphrag --ids $IDS --scope-sources \
+    --preset prod --retriever graphrag --ids $IDS --scope-sources --disable-kb \
+    --model google-genai:gemini-3.5-flash \
     --out runs/grag_graphrag
 # grade (auto retrieval/citation metrics, no judge needed) + report side by side
 uv run -m evals.grade --run runs/grag_classical

evals/part_c_plan.md

@@ -157,7 +157,7 @@ Same authoring model as v1 — real filler + authored planted facts + a binary `
 1. **Freeze + data discipline.** v2 is a new file; never touch v1. Gitignored; lives only in the private `ai-tutor-data` HF dataset under `eval/`.
 2. **Author Tier 1 first** (moderate contradiction sessions), reusing real corpus filler; model the shape on `data/eval/sessions_generated_*.jsonl` and the schema in `data/eval/README.md`. Add Tier 2 long-horizon sessions only if we decide Q1 is worth the spend/workshop story.
 3. **Calibrate by tokens and observed context state, not turn count.** For Q2, confirm prod summarization/compaction fired and A is no longer in the kept-recent window before the contradiction probe; if A is still visible to prod, the probe measures nothing. For Q1, smoke-run a few lengths (for example 30/45/60 turns) and confirm `full_history` cost actually diverges before running a screen.
-4. **Wire grading** — add `contradiction` / `longhorizon_recall` / `entity_isolation` rubric entries to `evals/judge.py` (`RUBRICS`) and the human rubric.
+4. **Wire grading** — add `contradiction` / `longhorizon_recall` / `entity_isolation` rubric entries to `evals/judge.py` (`RUBRICS`) and the human rubric. *(Done — all three are in `RUBRICS`.)*
 5. **Test `full_history` + `prod` + `incontext` + active `profile_memory` first.** This answers the cheap questions and tells us what, if anything, to build: does `full_history` fail contradictions (Q2), does `profile_memory` recover them through the store, and does `incontext` match `full_history`'s recall at lower all-in cost once sessions are actually long (Q1)?
 6. **Build only what the data demands**, one at a time, single-axis vs prod (each new mechanism needs its own `context_stats` signal mirrored in `evals/common.py` or the probe gate will not see it fire):
    - `temporal_graph_memory` — **only if** `full_history` fails Q2's contradictions.
@@ -171,7 +171,7 @@ Same authoring model as v1 — real filler + authored planted facts + a binary `
 - `runs/e2_v2_prod`: all 6 sessions, 158 turns, 0 errors, all probes under compaction, estimated chat cost **$50.47**. Handgraded partial result: **14% probe accuracy (1/7)**; contradiction **0/3**, long-horizon recall **0/2**, entity isolation **1/2**.
 - `runs/e2_v2_full_history`: 2 Tier-1 sessions, 44 turns, 0 errors, no compaction, estimated chat cost **$4.51**. Handgraded partial result: contradiction **2/2**.
 
-Interpretation: enough for a directional signal, not a final finding. The clean next run is **Tier 1 only**: finish the missing `full_history` contradiction session, then run Tier-1 `profile_memory` and optionally Tier-1 `incontext_history_retrieval`. That gives the useful Q2 answer with 3 probes/arm and avoids buying the long-horizon cost story before we know it matters.
+Interpretation: enough for a directional signal, not a final finding. The clean next run is **Tier 1 only**: finish the missing `full_history` contradiction session, then run Tier-1 `profile_memory` and optionally Tier-1 `incontext_history_retrieval`. That gives the useful Q2 answer with 3 probes/arm and avoids buying the long-horizon cost story before we know it matters. **(Superseded 2026-07-15: the DeepSeek stage-1 factorial ran the full v2 battery ×3 trials — evals.md F35–F38 — and `battery_sessions_v2.jsonl` was deprecated after a filler-confound audit; new runs use `battery_sessions_v2_1.jsonl`.)**
 
 ## Status / next actions
 
@@ -180,7 +180,7 @@ Interpretation: enough for a directional signal, not a final finding. The clean
   - Axis A (preset-shaped): `sliding_window`, `prompt_compression`, `selective_retention`, `context_reset`.
   - Axis A (subsystem): `incontext_history_retrieval` (turn-block retrieval over chat history; offline-tested with a stub embedder; embeds via Cohere at run time).
   - Tests: `tests/test_memory_variants.py` (+ turn-signal tests in `tests/test_telemetry.py`).
-- **Deferred / conditional:** `delta_summarization` needs a faithful append-only summary middleware and only becomes worth building after the long-horizon screen shows a real cost regime; `hierarchical_summarization` is dropped unless we later test extreme sessions; `collection_memory` is an optional v1/product follow-up for verbatim fact storage (F22 tested the current profile write-back, not that); `entity_memory` waits for multi-project probes to show bleed; `sleeptime_consolidation` remains deferred because it needs multi-thread data plus a between-session runner hook.
+- **Deferred / conditional:** `delta_summarization` and `hierarchical_summarization` *(status: since built as presets and screened — see `evals/compaction.md` and `evals/slm_compaction.md`)*; `collection_memory` is an optional v1/product follow-up for verbatim fact storage (F22 tested the current profile write-back, not that); `entity_memory` waits for multi-project probes to show bleed; `sleeptime_consolidation` remains deferred because it needs multi-thread data plus a between-session runner hook.
 - **SUBSET SCREEN DONE (2026-06-15).** All 11 arms run (~660 turns, 0 errors, ≈$166 at correct pricing), graded by the validated subagent judge, reported to `runs/c_report/report.md`. Findings F14–F24 recorded in `evals.md` (F21–F24 added later: `profile_memory`-on-sessions rerun (D), KB-fair recall regrade, and the `cleared_tool_outputs`-blind-signal correction). Headlines: F9/F10 reproduced (`full_history` cheapest + best memory); `incontext_history_retrieval` 83% but short-session-neutral (needs long sessions); `kb_off` inverts recall@shown 50→96% but priciest + worst memory; `retrieval_budget_30k` matches 100k recall; `observation_truncation` backfires; `context_reset`/`selective_retention`/`sliding_window` dominated. **Workshop-level Part C is complete.** Winners→promotion: `incontext_history_retrieval` (long-session test), `retrieval_budget_30k`, `clear_retrieval_kb`, anchor `full_history`.
 - **Beyond workshop (optional, gated — none needed for the workshop):** (1) **promotion** — 2-3 winners (+1-2 combos) × full batteries × 3 trials, hardens the rankings (doesn't answer anything new): **~$1,300-1,800 at corrected pricing**; (2) **Tier-1 contradiction v2** — now the priority path, because the full v2 screen is quota/time-heavy and Tier 1 already shows the strongest signal; (3) **the `incontext` long-session cost test** — principled but less product-critical unless real telemetry shows long sessions are common; (4) conditional builds (`temporal_graph_memory`, `delta_summarization`, `entity_memory`, optional `collection_memory`) only after the relevant baseline failure appears.
 - **Omar (open, non-blocking):** decide the `13167988` v2 reword; sign off the review-log audit; push `a564460` + the Part C/cost-fix working-tree changes; greenlight promotion spend / author `_v2` only if going beyond the workshop.

evals/slm_compaction.md

@@ -2,9 +2,9 @@
 
 Two scoped experiments on **how to survive a long lesson in a small context
 window**, run on cheap **local** models (Ollama) instead of a large cloud model,
-across three SLMs that run on a 16 GB Mac. It lives entirely on branch
-`experiment/slm-compaction` and is fully opt-in: the default chat model and all
-production behavior are unchanged.
+across three SLMs that run on a 16 GB Mac. It was developed on branch
+`experiment/slm-compaction` (since merged) and is fully opt-in: the default chat
+model and all production behavior are unchanged.
 
 ## Why this exists (the gap it fills)