feat: add findings on DeepSeek-V4-Flash performance and memory strategies

evals.md

@@ -168,14 +168,18 @@ Numbered findings; each states what it was tested on. Convention: entries are ne
 
 - **F24 (2026-06-19) — F3's "0 clears" was a blind-signal artifact: `cleared_tool_outputs` cannot observe `ClearToolUsesEdit`, which fires but invisibly.** `ContextEditingMiddleware.wrap_model_call` applies the clear to a `deepcopy` of the per-call message view and returns it via `request.override(messages=...)` (`langchain/agents/middleware/context_editing.py:251-255`) — it never writes the placeholder back to the checkpoint, but `telemetry.context_window_stats` counts placeholders in the *checkpointed* `state_messages` (`chat_service.py:1876`). So the counter reads **0 for every preset including `aggressive`** across all `runs/`, regardless of whether clearing fired — "0 clears" is structural, not evidence of absence. The real effect lands in input tokens: on sessions `clear_retrieval_kb` cut mean input **145.6k→120.5k (~17%, $0.229→$0.203/turn)** vs `prod` by clearing the retrieval payloads `prod` exempts, while on single-turn it **backfired to 308k/$0.477** (re-retrieval churn to recover what it cut — the `aggressive`/F19 pathology). So clearing is real and *mixed* (session win, single-turn loss), not absent. **Supersedes F3's "clearing never fires" reading; F3's structural point stands** — `prod`'s `exclude_tools=("retrieve_tutor_context",)` provably leaves the dominant retrieval payloads in place by construction, so prod-clearing still can't touch the big tokens. Caveats: the session token saving is 1-trial screen n; library clearing stays telemetry-invisible (the probe gate survives only because summarization, which *does* persist, co-fires in these arms — a clearing-only arm would false-fail `check_triggers`). Detect clearing via input-token deltas, not `cleared_tool_outputs`.
 
+- **F25 (2026-06-19) — DeepSeek-V4-Flash replicates F9/F10 and extends them to long horizon: `full_history` is cheapest AND best-memory on every tier, so the cost crossover never appears.** A full Part-C-style v2 matrix re-run on **DeepSeek-V4-Flash (first-party API, not Gemini)** — 12 arms, 840 turns, 0 errors; Tier 1 contradiction (3×22-turn) + Tier 2 long-horizon (2×36-turn); reports `runs/ds_v2_t{1,2}_report/`, raw on HF `eval_runs/part_f_deepseek_v2`. `full_history` is the **cheapest** arm on both tiers (**$0.0069/turn** at 22t, **$0.0101/turn** at 36t), undercutting every compaction arm — while billing the *most* tokens (1.78M input tok/turn at long horizon) because ~**97% are cache-hit**. DeepSeek's ~50× cache discount ($0.14 cache-miss vs $0.0028 cache-hit input) makes the enormous *cached* prefix cheaper than any compaction's cache-breaking rewrite, so the summarization arms are priciest (`prod` $0.0336, `selective_retention` $0.0357). F9's inversion therefore holds **even at 36 turns**, and harder than on Gemini (10× discount → crossover would appear sooner; DeepSeek's 50× pushes it past every length tested). Per-turn cost is ~10-15× below Gemini on the comparable tier. **Provider caveat:** `incontext_history_retrieval` does *not* beat `full_history` on cost here ($0.0147 vs $0.0101) — the opposite of a teammate's OpenRouter run, where fallback routing fragments the prefix cache; cost comparisons must hold the provider/caching path constant (first-party single-endpoint caching verified live, `cache_read` non-zero). Caveats: thin n (Tier 1 n=3, Tier 2 n=2 per arm, 1 trial) — coarse rankings.
+
+- **F26 (2026-06-19) — The v2 contradiction question (Q2) is answered on DeepSeek: `full_history` resolves contradictions, compaction loses them, so no temporal memory is needed.** Same matrix, judge-graded via the validated **subagent path** (28 probes, blind, `judge.py` rubric — F14). `full_history` scores **3/3 contradiction** and **2/2 long-horizon recall** — perfect memory on both tiers, including the regime where it could in theory lose (it carries both A and A′ raw and must pick the current A′). `prod` summarization **fails contradictions 0/3** (evicts the updated fact — F10 confirmed and sharpened) and 1/2 long-horizon. `incontext_history_retrieval` matches `full_history` (3/3, 2/2) by retrieving the old turns, but costs more (F25); `profile_memory` partially rescues contradictions (2/3) via its store (the F22 pattern). Among long-horizon compaction arms, `prompt_compression` is least-bad (2/2 — shrinks text, drops no facts) and `context_reset` is dominated (0/2). Validity: compaction fired at **100% of probes** for every compaction arm, **0%** for `full_history` (gate passed). **Resolves item 4's conditional — `temporal_graph_memory` is not triggered**, since `full_history` did not fail contradictions. Caveats: thin n (3/2 per arm, 1 trial); the judge is reproducibility-validated (F14), not independent-human.
+
 **Screen winners → promotion candidates:** `incontext_history_retrieval` (needs the long-session test), `retrieval_budget_30k` and `clear_retrieval_kb` (cheap Axis-B wins; clear_retrieval_kb is cheaper than prod with similar memory + better recall/key-points), anchored by `full_history`. **Drop:** `context_reset`, `observation_truncation`, `selective_retention`.
 
 **Part C — what remains (beyond the workshop).** The screen answered the core question; everything below *hardens* or *extends* it and is **gated on a spend or data decision from Omar** — none is required for the workshop.
 
 1. **Promotion (rigor, not new findings).** Re-run the 2-3 winners (+ 1-2 combos) on the full batteries (60 single-turn + 32 sessions + 30 replay) × 3 trials, with paired statistical tests and per-variant failure-taxonomy diffs, to turn the screen's coarse n=12 rankings into confident numbers; optionally an Anthropic (Haiku) re-run of the prefix-rewriting arms (`prompt_compression`, `context_reset`, `aggressive`) to test whether explicit caching flips F2's cost ranking. **~$1,300-1,800 at corrected pricing** (the old "$300-400" was at pre-correction prices; × the 4.4 fix — estimate precisely before running, since the per-turn cost is now ~$0.25).
-2. **Finish the v2 contradiction tier — the highest-value new dataset (already built and partially run; see the V2 execution note below).** Moderate sessions: plant A, update to A′, then probe after prod compaction has evicted A from the kept-recent window. This is the one correctness regime where `full_history` itself might lose (it sees both A and A′ and must choose the current one) — yet the partial run so far shows the opposite (`full_history` contradiction **2/2** vs `prod` **0/3**), so that regime has not actually appeared. What remains is finishing Tier 1 (the missing `full_history` session, then Tier-1 `profile_memory` and optionally `incontext_history_retrieval`) and adding trials, not building the dataset.
-3. **The `incontext_history_retrieval` long-session cost test — principled but less product-critical.** The screen showed `incontext` works (83%, F16) but couldn't show a cost benefit on ≤13-turn sessions. Whether retrieve-old-turns beats `full_history` *once sessions get genuinely long* needs a token-calibrated long-session v2 tier; useful for the workshop argument, but gated by whether that regime matters in real tutor telemetry.
-4. **Conditional builds.** Build `temporal_graph_memory` only if `full_history` fails contradictions, `delta_summarization` only if the long-horizon cost tier shows room to beat full history, and `entity_memory` only if multi-project probes show fact bleed. `collection_memory` is a separate cheap v1/product follow-up for verbatim fact storage (F22 tested the current profile write-back, not that). `hierarchical_summarization` and `sleeptime_consolidation` are deferred. The step-by-step v2 plan is in `evals_part_c_plan.md` → "Battery v2".
+2. **Finish the v2 contradiction tier — the highest-value new dataset (already built and partially run; see the V2 execution note below).** Moderate sessions: plant A, update to A′, then probe after prod compaction has evicted A from the kept-recent window. This is the one correctness regime where `full_history` itself might lose (it sees both A and A′ and must choose the current one) — yet the partial run so far shows the opposite (`full_history` contradiction **2/2** vs `prod` **0/3**), so that regime has not actually appeared. What remains is finishing Tier 1 (the missing `full_history` session, then Tier-1 `profile_memory` and optionally `incontext_history_retrieval`) and adding trials, not building the dataset. **(2026-06-19 — now run in full on DeepSeek-V4-Flash rather than Gemini: F26 confirms `full_history` 3/3 contradiction vs `prod` 0/3, so the regime did not appear on a second model either; the Gemini Tier-1 finish is now optional cross-model confirmation.)**
+3. **The `incontext_history_retrieval` long-session cost test — principled but less product-critical.** The screen showed `incontext` works (83%, F16) but couldn't show a cost benefit on ≤13-turn sessions. Whether retrieve-old-turns beats `full_history` *once sessions get genuinely long* needs a token-calibrated long-session v2 tier; useful for the workshop argument, but gated by whether that regime matters in real tutor telemetry. **(2026-06-19 — run on DeepSeek (F25) at 36 turns: `full_history` stays cheapest and `incontext` works but does not beat it on cost under first-party caching; whether the answer flips on a 10× cache provider like Gemini/Anthropic is still open.)**
+4. **Conditional builds.** Build `temporal_graph_memory` only if `full_history` fails contradictions (F26: it did **not** on DeepSeek — 3/3 — so this stays unbuilt), `delta_summarization` only if the long-horizon cost tier shows room to beat full history (F25: `full_history` still cheapest at 36t on DeepSeek, so no room there — though a 10× cache provider is untested), and `entity_memory` only if multi-project probes show fact bleed. `collection_memory` is a separate cheap v1/product follow-up for verbatim fact storage (F22 tested the current profile write-back, not that). `hierarchical_summarization` and `sleeptime_consolidation` are deferred. The step-by-step v2 plan is in `evals_part_c_plan.md` → "Battery v2".
 
 **Deliberately not pursued.** *Stretch (heavier, likely later):* sub-agent isolation, reframed as a tool-output-token play — keep noisy `run_kb_command` output out of the main context (ties to F1); temporal-graph memory is already the conditional build in item 4 above. *Dropped as low-information given the findings:* the skills / lazy-prompt-loading family (only a ~458-token win — see the talk-outline correction below), procedural memory, GraphRAG, and multi-agent / parallel-research agents. Context for the `kb_off` arm (F17/F23): the KB is the dominant grounding tool — ≈89% of Part B turns, ~7.7 KB vs ~0.9 retrieval calls/turn over 1,088 turns — so disabling it forces the 100k-token retrieval fallback, which is why it raised cost/latency.
 
@@ -219,3 +223,4 @@ Golden cases as a CI gate (5–10 critical-path cases with deterministic asserti
 - `data/eval/README.md` — battery schemas + glossary of terms; `review_log_v1.md` — dataset audit trail.
 - `evals/` — the harness code; `app/memory_presets.py`, `app/telemetry.py` — the app-side hooks.
 - `runs/b_report/`, `runs/c_report/` — generated tables, token curves, blinded grading workbook (Part B and the Part C screen). Re-grade / follow-up reports: `runs/kbfair_report/` (F23 KB-fair recall), `runs/d_report_profile_memory_active/` (F22 profile rerun), `runs/e2_v2_partial_report/` (v2 partial).
+- `runs/ds_v2_t1_report/`, `runs/ds_v2_t2_report/` — **DeepSeek-V4-Flash** v2 matrix (F25/F26: cost + contradiction + long-horizon), graded via the subagent path (`runs/ds_judge/`). Shared on HF as `eval_runs/part_f_deepseek_v2/` (the standard `mv eval_runs/part_*/* runs/` download restores them). Provider added in `app/chat_service.py` (`deepseek` / `openrouter`), pricing in `app/telemetry.py`.