chopratejas commited on
Commit
c14b9ac
·
1 Parent(s): 2bd9316

Add headroom learn: offline failure learning for coding agents

Browse files

Analyzes past conversation history to find tool call failure patterns,
correlates each failure with what eventually succeeded, and writes
specific project-level learnings to CLAUDE.md and MEMORY.md.

Key design:
- Success correlation: extracts the diff between failed and successful
inputs as the learning (not generic advice)
- Generic architecture: tool-agnostic ToolCall model with pluggable
Scanner/Writer adapters (Claude Code first, extensible to Cursor/Codex)
- 5 analyzers: Environment, Structure, Commands, Retries, Cross-Session
- Dry-run by default, --apply to write, --all for all projects

Also fixes mypy errors in litellm_callback, asgi, langchain chat_model,
and anthropic provider (AsyncClient typing, ToolCall arg-type, int cast).

CHANGELOG.md CHANGED
@@ -8,6 +8,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
8
  ## [Unreleased]
9
 
10
  ### Added
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  - **any-llm backend** - Route requests through 38+ LLM providers (OpenAI, Mistral, Groq, Ollama, etc.) via [any-llm](https://mozilla-ai.github.io/any-llm/providers/)
12
  - Enable with `--backend anyllm --anyllm-provider <provider>`
13
  - Install with: `pip install 'headroom-ai[anyllm]'`
 
8
  ## [Unreleased]
9
 
10
  ### Added
11
+ - **`headroom learn`** — Offline failure learning for coding agents
12
+ - Analyzes past conversation history (Claude Code, extensible to Cursor/Codex)
13
+ - **Success correlation**: for each failure, finds what succeeded after and extracts the specific correction
14
+ - 5 analyzers: Environment, Structure, Command Patterns, Retry Prevention, Cross-Session
15
+ - Writes specific learnings to CLAUDE.md (stable project facts) and MEMORY.md (session patterns)
16
+ - Generic architecture: tool-agnostic `ToolCall` model, pluggable Scanner/Writer adapters
17
+ - Dry-run by default, `--apply` to write, `--all` for all projects
18
+ - Example output: "FirstClassEntity.java is not at axion-formats/ — actually at axion-scala-common/"
19
+ - **Read Lifecycle Management** — Event-driven compression of stale/superseded Read outputs
20
+ - Detects when a Read output becomes stale (file was edited after) or superseded (file was re-read)
21
+ - Replaces stale/superseded content with compact CCR markers, stores originals for retrieval
22
+ - 75% of Read output bytes are provably stale or redundant (from real-world analysis of 66K tool calls)
23
+ - Fresh Reads (latest read, no subsequent edit) are never touched — Edit safety preserved
24
+ - Opt-in via `ReadLifecycleConfig(enabled=True)`, disabled by default
25
+ - Handles both OpenAI and Anthropic message formats
26
  - **any-llm backend** - Route requests through 38+ LLM providers (OpenAI, Mistral, Groq, Ollama, etc.) via [any-llm](https://mozilla-ai.github.io/any-llm/providers/)
27
  - Enable with `--backend anyllm --anyllm-provider <provider>`
28
  - Install with: `pip install 'headroom-ai[anyllm]'`
README.md CHANGED
@@ -61,6 +61,16 @@ OPENAI_BASE_URL=http://localhost:8787/v1 cursor
61
 
62
  Works with any language, any tool, any framework. One env var. **[Proxy docs](docs/proxy.md)**
63
 
 
 
 
 
 
 
 
 
 
 
64
  ### Python: One function
65
 
66
  ```python
@@ -231,6 +241,8 @@ flowchart TB
231
  | **Image Compression** | 40-90% token reduction via trained ML router |
232
  | **Memory** | Persistent memory across conversations |
233
  | **Compression Hooks** | Customize compression with pre/post hooks |
 
 
234
 
235
  ---
236
 
@@ -276,6 +288,7 @@ Python 3.10+
276
  | [Memory](docs/memory.md) | Persistent memory |
277
  | [Agno](docs/agno.md) | Agno agent framework |
278
  | [MCP](docs/mcp.md) | Claude Code subscriptions |
 
279
  | [Configuration](docs/configuration.md) | All options |
280
 
281
  ---
 
61
 
62
  Works with any language, any tool, any framework. One env var. **[Proxy docs](docs/proxy.md)**
63
 
64
+ ### Failure Learning (new)
65
+
66
+ ```bash
67
+ headroom learn # Analyze past Claude Code sessions, show recommendations
68
+ headroom learn --apply # Write learnings to CLAUDE.md and MEMORY.md
69
+ headroom learn --all --apply # Learn across all your projects
70
+ ```
71
+
72
+ Reads your conversation history, finds every failed tool call, correlates it with what eventually succeeded, and writes specific corrections into your project files. Next session starts smarter. **[Learn docs](docs/learn.md)**
73
+
74
  ### Python: One function
75
 
76
  ```python
 
241
  | **Image Compression** | 40-90% token reduction via trained ML router |
242
  | **Memory** | Persistent memory across conversations |
243
  | **Compression Hooks** | Customize compression with pre/post hooks |
244
+ | **Read Lifecycle** | Detects stale/superseded Read outputs, replaces with CCR markers |
245
+ | **`headroom learn`** | Analyzes past failures, writes project-specific learnings to CLAUDE.md/MEMORY.md |
246
 
247
  ---
248
 
 
288
  | [Memory](docs/memory.md) | Persistent memory |
289
  | [Agno](docs/agno.md) | Agno agent framework |
290
  | [MCP](docs/mcp.md) | Claude Code subscriptions |
291
+ | [Learn](docs/learn.md) | Offline failure learning for coding agents |
292
  | [Configuration](docs/configuration.md) | All options |
293
 
294
  ---
docs/learn.md ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Headroom Learn
2
+
3
+ Offline failure learning for coding agents. Analyzes past conversations, finds what went wrong, correlates it with what eventually worked, and writes specific project-level learnings that prevent the same mistakes next session.
4
+
5
+ ## Quick Start
6
+
7
+ ```bash
8
+ # See recommendations for current project (dry-run, no changes)
9
+ headroom learn
10
+
11
+ # Write recommendations to CLAUDE.md and MEMORY.md
12
+ headroom learn --apply
13
+
14
+ # Analyze a specific project
15
+ headroom learn --project ~/my-project --apply
16
+
17
+ # Analyze all projects
18
+ headroom learn --all --apply
19
+ ```
20
+
21
+ ## How It Works
22
+
23
+ ```
24
+ Past Sessions → Scanner → Analyzer → Writer → CLAUDE.md / MEMORY.md
25
+ │ │ │
26
+ │ │ └─ Writes marker-delimited sections
27
+ │ │ (replaced on re-run, not duplicated)
28
+ │ │
29
+ │ └─ Success Correlation: for each failure,
30
+ │ finds what succeeded and extracts the diff
31
+
32
+ └─ Reads ~/.claude/projects/*.jsonl
33
+ (extensible to Cursor, Codex, etc.)
34
+ ```
35
+
36
+ ### Success Correlation
37
+
38
+ The core innovation. Instead of cataloging failures ("Read failed 5 times"), Headroom finds what the model did to fix each failure:
39
+
40
+ - **Failed**: `Read axion-formats/src/main/java/.../FirstClassEntity.java`
41
+ - **Then succeeded**: `Read axion-scala-common/src/main/scala/.../FirstClassEntity.scala`
42
+ - **Learning**: "`FirstClassEntity` is at `axion-scala-common/`, not `axion-formats/`"
43
+
44
+ This produces specific, actionable corrections — not generic advice.
45
+
46
+ ## What It Learns
47
+
48
+ ### 1. Environment Facts → CLAUDE.md
49
+ Which runtime commands work vs fail.
50
+
51
+ ```markdown
52
+ ### Environment
53
+ - **Python**: use `uv run python` (not `python3` — modules not available outside venv)
54
+ ```
55
+
56
+ ### 2. File Path Corrections → CLAUDE.md
57
+ Wrong paths the model keeps guessing, with the correct locations.
58
+
59
+ ```markdown
60
+ ### File Path Corrections
61
+ - `axion-common/src/.../AxionSparkConstants.scala`
62
+ → actually at `axion-spark-common/src/.../AxionSparkConstants.scala`
63
+ ```
64
+
65
+ ### 3. Search Scope → CLAUDE.md
66
+ Which directories to search in (narrow paths fail, broader ones work).
67
+
68
+ ```markdown
69
+ ### Search Scope
70
+ - Don't search `axion-model/` → use `axion/` (the repo root)
71
+ ```
72
+
73
+ ### 4. Command Patterns → CLAUDE.md
74
+ How commands should (and shouldn't) be run.
75
+
76
+ ```markdown
77
+ ### Command Patterns
78
+ - **user_prefers_manual**: User rejected gradle 18 times — show the command, don't execute
79
+ - **python_runtime**: Use `uv run python` not `python3` (ModuleNotFoundError)
80
+ ```
81
+
82
+ ### 5. Known Large Files → CLAUDE.md
83
+ Files that need `offset`/`limit` with Read.
84
+
85
+ ```markdown
86
+ ### Known Large Files
87
+ - `proxy/server.py` (~8000 lines) — always use offset/limit
88
+ ```
89
+
90
+ ### 6. Retry Prevention → MEMORY.md
91
+ Specific suggestions derived from actual corrections.
92
+
93
+ ### 7. Permission Notes → MEMORY.md
94
+ Commands repeatedly rejected — model should suggest them to the user instead.
95
+
96
+ ## Where Learnings Go
97
+
98
+ | Pattern | Destination | Why |
99
+ |---------|-------------|-----|
100
+ | Environment, paths, search scope, commands, large files | **CLAUDE.md** | Stable project facts, version-controllable |
101
+ | Missing paths, retry patterns, permissions | **MEMORY.md** | May change, agent-specific |
102
+
103
+ CLAUDE.md lives in your project directory. MEMORY.md lives in `~/.claude/projects/*/memory/`.
104
+
105
+ ## Marker-Based Updates
106
+
107
+ Headroom manages a clearly-delimited section in each file:
108
+
109
+ ```markdown
110
+ <!-- headroom:learn:start -->
111
+ ## Headroom Learned Patterns
112
+ *Auto-generated by `headroom learn` — do not edit manually*
113
+ ...
114
+ <!-- headroom:learn:end -->
115
+ ```
116
+
117
+ On re-run, only the content between markers is replaced. Your existing file content is preserved.
118
+
119
+ ## Architecture
120
+
121
+ ```
122
+ Scanner (adapter) → Analyzer (generic) → Writer (adapter)
123
+ ├── ClaudeCodeScanner ├── EnvironmentAnalyzer ├── ClaudeCodeWriter
124
+ ├── (CursorScanner) ├── StructureAnalyzer ├── (CursorWriter)
125
+ └── (GenericScanner) ├── CommandAnalyzer └── (GenericWriter)
126
+ ├── RetryAnalyzer
127
+ └── CrossSessionAnalyzer
128
+ ```
129
+
130
+ **Scanners** read tool-specific log formats and produce normalized `ToolCall` sequences.
131
+ **Analyzers** work on `ToolCall` — same analysis for any agent system.
132
+ **Writers** output to tool-specific context injection mechanisms.
133
+
134
+ To add support for a new agent (e.g., Cursor):
135
+ 1. Write `CursorScanner(ConversationScanner)` — reads Cursor's log format
136
+ 2. Write `CursorWriter(ContextWriter)` — writes to `.cursorrules`
137
+ 3. Same analyzers, same models, same recommendations
138
+
139
+ ## CLI Reference
140
+
141
+ ```
142
+ headroom learn [OPTIONS]
143
+
144
+ Options:
145
+ --project PATH Project directory to analyze (default: current directory)
146
+ --all Analyze all discovered projects
147
+ --apply Write recommendations (default: dry-run)
148
+ --claude-dir PATH Path to .claude directory (default: ~/.claude)
149
+ ```
150
+
151
+ ## Real-World Results
152
+
153
+ Tested on 67,583 tool calls across 23 projects:
154
+
155
+ | Metric | Value |
156
+ |--------|-------|
157
+ | Failure rate | 7.5% (5,066 failures) |
158
+ | Corrections extracted | 164 per project (avg) |
159
+ | Specific path corrections | 22 (axion project) |
160
+ | Search scope corrections | 24 (axion project) |
161
+ | Command patterns learned | 5 (axion project) |
162
+ | Estimated preventable waste | ~27 MB across corpus |
headroom/cli/learn.py ADDED
@@ -0,0 +1,148 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """CLI commands for Headroom Learn — offline failure learning."""
2
+
3
+ from __future__ import annotations
4
+
5
+ from pathlib import Path
6
+
7
+ import click
8
+
9
+ from .main import main
10
+
11
+
12
+ @main.command()
13
+ @click.option(
14
+ "--project",
15
+ type=click.Path(exists=True, path_type=Path),
16
+ default=None,
17
+ help="Project directory to analyze. Defaults to current directory.",
18
+ )
19
+ @click.option(
20
+ "--all",
21
+ "analyze_all",
22
+ is_flag=True,
23
+ default=False,
24
+ help="Analyze all discovered projects.",
25
+ )
26
+ @click.option(
27
+ "--apply",
28
+ is_flag=True,
29
+ default=False,
30
+ help="Write recommendations to CLAUDE.md / MEMORY.md (default: dry-run).",
31
+ )
32
+ @click.option(
33
+ "--claude-dir",
34
+ type=click.Path(path_type=Path),
35
+ default=None,
36
+ help="Path to .claude directory. Defaults to ~/.claude.",
37
+ )
38
+ def learn(
39
+ project: Path | None,
40
+ analyze_all: bool,
41
+ apply: bool,
42
+ claude_dir: Path | None,
43
+ ) -> None:
44
+ """Learn from past tool call failures to prevent future ones.
45
+
46
+ Analyzes conversation history to find failure patterns (wrong paths,
47
+ missing modules, stubborn retries) and generates context that prevents
48
+ them from recurring.
49
+
50
+ \b
51
+ Examples:
52
+ headroom learn # Analyze current project (dry-run)
53
+ headroom learn --apply # Write recommendations
54
+ headroom learn --all # Analyze all projects
55
+ headroom learn --project ~/myapp # Analyze specific project
56
+ """
57
+ from ..learn.analyzer import FailureAnalyzer
58
+ from ..learn.scanner import ClaudeCodeScanner
59
+ from ..learn.writer import ClaudeCodeWriter, Recommender
60
+
61
+ scanner = ClaudeCodeScanner(claude_dir=claude_dir)
62
+ analyzer = FailureAnalyzer()
63
+ recommender = Recommender()
64
+ writer = ClaudeCodeWriter()
65
+
66
+ # Discover projects
67
+ all_projects = scanner.discover_projects()
68
+
69
+ if not all_projects:
70
+ click.echo("No projects found in ~/.claude/projects/")
71
+ return
72
+
73
+ # Filter to target project(s)
74
+ if analyze_all:
75
+ targets = all_projects
76
+ elif project:
77
+ resolved = project.resolve()
78
+ targets = [p for p in all_projects if p.project_path == resolved]
79
+ if not targets:
80
+ click.echo(f"Project not found: {resolved}")
81
+ click.echo(f"Available projects: {', '.join(p.name for p in all_projects)}")
82
+ return
83
+ else:
84
+ # Auto-detect from cwd
85
+ cwd = Path.cwd().resolve()
86
+ targets = [p for p in all_projects if p.project_path == cwd]
87
+ if not targets:
88
+ # Try parent directories
89
+ for parent in cwd.parents:
90
+ targets = [p for p in all_projects if p.project_path == parent]
91
+ if targets:
92
+ break
93
+ if not targets:
94
+ click.echo(f"No project data found for {cwd}")
95
+ click.echo("Try: headroom learn --project <path> or headroom learn --all")
96
+ click.echo("\nAvailable projects:")
97
+ for p in all_projects[:10]:
98
+ click.echo(f" {p.name:30s} {p.project_path}")
99
+ return
100
+
101
+ # Analyze each target
102
+ for proj in targets:
103
+ click.echo(f"\n{'=' * 60}")
104
+ click.echo(f"Project: {proj.name}")
105
+ click.echo(f"Path: {proj.project_path}")
106
+ click.echo(f"{'=' * 60}")
107
+
108
+ sessions = scanner.scan_project(proj)
109
+ if not sessions:
110
+ click.echo(" No conversation data found.")
111
+ continue
112
+
113
+ report = analyzer.analyze(proj, sessions)
114
+
115
+ # Print summary
116
+ click.echo(f"\n Sessions analyzed: {report.total_sessions}")
117
+ click.echo(f" Total tool calls: {report.total_calls}")
118
+ click.echo(f" Failed calls: {report.total_failures} ({report.failure_rate:.1%})")
119
+ click.echo(f" Waste bytes: {report.waste_bytes / 1024:.0f} KB")
120
+
121
+ if report.failure_rate == 0:
122
+ click.echo("\n No failures found. Nothing to learn.")
123
+ continue
124
+
125
+ # Generate recommendations
126
+ recommendations = recommender.recommend(report)
127
+
128
+ if not recommendations:
129
+ click.echo("\n No actionable patterns found.")
130
+ continue
131
+
132
+ click.echo(f"\n Recommendations: {len(recommendations)}")
133
+
134
+ # Write (or dry-run)
135
+ result = writer.write(recommendations, proj, dry_run=not apply)
136
+
137
+ for file_path, content in result.content_by_file.items():
138
+ click.echo(f"\n {'[WOULD WRITE]' if result.dry_run else '[WROTE]'} {file_path}")
139
+ click.echo(f" {'─' * 50}")
140
+ # Show content preview (indented)
141
+ for line in content.split("\n"):
142
+ if line.startswith("<!-- headroom"):
143
+ continue # Skip markers in display
144
+ click.echo(f" {line}")
145
+ click.echo(f" {'─' * 50}")
146
+
147
+ if result.dry_run:
148
+ click.echo("\n Dry run — no files modified. Use --apply to write.")
headroom/cli/main.py CHANGED
@@ -35,6 +35,7 @@ def _register_commands() -> None:
35
  """Register all subcommand groups."""
36
  from . import (
37
  evals, # noqa: F401
 
38
  mcp, # noqa: F401
39
  memory, # noqa: F401
40
  proxy, # noqa: F401
 
35
  """Register all subcommand groups."""
36
  from . import (
37
  evals, # noqa: F401
38
+ learn, # noqa: F401
39
  mcp, # noqa: F401
40
  memory, # noqa: F401
41
  proxy, # noqa: F401
headroom/integrations/asgi.py CHANGED
@@ -80,7 +80,7 @@ class CompressionMiddleware:
80
  self._api_url = (
81
  api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
82
  ).rstrip("/")
83
- self._client = None # Lazy-initialized httpx.AsyncClient
84
 
85
  @property
86
  def cloud_mode(self) -> bool:
 
80
  self._api_url = (
81
  api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
82
  ).rstrip("/")
83
+ self._client: Any = None # Lazy-initialized httpx.AsyncClient
84
 
85
  @property
86
  def cloud_mode(self) -> bool:
headroom/integrations/langchain/chat_model.py CHANGED
@@ -79,7 +79,7 @@ def _check_langchain_available() -> None:
79
  )
80
 
81
 
82
- def _tool_call_args_to_json(tc: dict[str, Any]) -> str:
83
  """Normalize tool call arguments to JSON string for OpenAI format.
84
 
85
  LangChain can provide 'args' (dict) or 'arguments' (str) depending on source.
 
79
  )
80
 
81
 
82
+ def _tool_call_args_to_json(tc: dict[str, Any] | Any) -> str:
83
  """Normalize tool call arguments to JSON string for OpenAI format.
84
 
85
  LangChain can provide 'args' (dict) or 'arguments' (str) depending on source.
headroom/integrations/litellm_callback.py CHANGED
@@ -69,7 +69,7 @@ class HeadroomCallback:
69
  self._api_url = (
70
  api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
71
  ).rstrip("/")
72
- self._client = None # Lazy-initialized httpx.AsyncClient
73
 
74
  @property
75
  def total_tokens_saved(self) -> int:
 
69
  self._api_url = (
70
  api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
71
  ).rstrip("/")
72
+ self._client: Any = None # Lazy-initialized httpx.AsyncClient
73
 
74
  @property
75
  def total_tokens_saved(self) -> int:
headroom/learn/__init__.py ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Headroom Learn — offline failure learning for coding agents.
2
+
3
+ Analyzes conversation logs to find tool call failure patterns and generates
4
+ context (CLAUDE.md, MEMORY.md, .cursorrules, etc.) that prevents future failures.
5
+
6
+ Architecture:
7
+ Scanner (adapter) → Analyzer (generic) → Writer (adapter)
8
+ ├── ClaudeCodeScanner ├── EnvironmentAnalyzer ├── ClaudeCodeWriter
9
+ ├── CursorScanner ├── StructureAnalyzer ├── CursorWriter
10
+ └── GenericScanner ├── RetryAnalyzer └── GenericWriter
11
+ ├── PermissionAnalyzer
12
+ └── CrossSessionAnalyzer
13
+
14
+ Scanners read tool-specific log formats and produce normalized ToolCall sequences.
15
+ Analyzers work on ToolCall — same analysis for any agent system.
16
+ Writers output to tool-specific context injection mechanisms.
17
+ """
headroom/learn/analyzer.py ADDED
@@ -0,0 +1,618 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Failure analyzers with success correlation.
2
+
3
+ The core insight: don't just catalog failures — find what SUCCEEDED after
4
+ each failure. The diff between failed input and successful input is the
5
+ actual learning.
6
+
7
+ All analyzers work on normalized ToolCall sequences. They are tool-agnostic:
8
+ same analysis works for Claude Code, Cursor, Codex, or any agent with tool calls.
9
+ """
10
+
11
+ from __future__ import annotations
12
+
13
+ import os
14
+ import re
15
+ from collections import Counter, defaultdict
16
+
17
+ from .models import (
18
+ AnalysisReport,
19
+ CommandPattern,
20
+ Correction,
21
+ EnvironmentFact,
22
+ ErrorCategory,
23
+ ProjectInfo,
24
+ RetryPattern,
25
+ SessionData,
26
+ StructureNote,
27
+ ToolCall,
28
+ )
29
+
30
+ # How many messages ahead to look for a success after a failure
31
+ _CORRECTION_WINDOW = 10
32
+
33
+
34
+ class FailureAnalyzer:
35
+ """Runs all analyzers on tool call data and produces an AnalysisReport."""
36
+
37
+ def analyze(self, project: ProjectInfo, sessions: list[SessionData]) -> AnalysisReport:
38
+ all_calls = [tc for s in sessions for tc in s.tool_calls]
39
+ failed_calls = [tc for tc in all_calls if tc.is_error]
40
+
41
+ report = AnalysisReport(
42
+ project=project,
43
+ total_calls=len(all_calls),
44
+ total_failures=len(failed_calls),
45
+ total_sessions=len(sessions),
46
+ waste_bytes=sum(tc.output_bytes for tc in failed_calls),
47
+ )
48
+
49
+ # Phase 1: Extract failure→success corrections (the core learning)
50
+ report.corrections = _extract_corrections(sessions)
51
+
52
+ # Phase 2: Analyze specific dimensions using corrections + raw data
53
+ report.environment_facts = _analyze_environment(sessions)
54
+ report.structure_notes = _analyze_structure(sessions, report.corrections)
55
+ report.command_patterns = _analyze_commands(sessions, report.corrections)
56
+ report.retry_patterns = _analyze_retries(sessions, report.corrections)
57
+ report.permission_issues = _analyze_permissions(sessions)
58
+ report.cross_session_patterns = _analyze_cross_session(sessions)
59
+
60
+ return report
61
+
62
+
63
+ # =============================================================================
64
+ # Success Correlation: The Core Learning Primitive
65
+ # =============================================================================
66
+
67
+
68
+ def _extract_corrections(sessions: list[SessionData]) -> list[Correction]:
69
+ """For each failure, find the next success of the same tool type.
70
+
71
+ The pair (failed_input, success_input) is a Correction — the model
72
+ learned something and corrected its approach.
73
+ """
74
+ corrections: list[Correction] = []
75
+
76
+ for session in sessions:
77
+ calls = session.tool_calls
78
+ for i, tc in enumerate(calls):
79
+ if not tc.is_error:
80
+ continue
81
+ # Skip sibling errors (cascades, not real failures)
82
+ if tc.error_category == ErrorCategory.SIBLING_ERROR:
83
+ continue
84
+ # Look forward for a success of the same tool
85
+ for j in range(i + 1, min(i + _CORRECTION_WINDOW, len(calls))):
86
+ candidate = calls[j]
87
+ if candidate.name != tc.name or candidate.is_error:
88
+ continue
89
+ # Found a success — is the input meaningfully different?
90
+ if candidate.input_data == tc.input_data:
91
+ continue # Exact same input succeeded (transient error)
92
+ corrections.append(
93
+ Correction(
94
+ tool_name=tc.name,
95
+ failed_input=tc.input_data,
96
+ success_input=candidate.input_data,
97
+ error_category=tc.error_category,
98
+ session_id=session.session_id,
99
+ )
100
+ )
101
+ break
102
+
103
+ return corrections
104
+
105
+
106
+ # =============================================================================
107
+ # Environment Analyzer (uses corrections for python/build tool detection)
108
+ # =============================================================================
109
+
110
+
111
+ def _analyze_environment(sessions: list[SessionData]) -> list[EnvironmentFact]:
112
+ """Detect which runtime commands work vs fail."""
113
+ facts: list[EnvironmentFact] = []
114
+
115
+ python_failures: Counter[str] = Counter()
116
+ python_successes: Counter[str] = Counter()
117
+ python_sessions: dict[str, set[str]] = defaultdict(set)
118
+
119
+ for session in sessions:
120
+ for tc in session.tool_calls:
121
+ if tc.name not in ("Bash", "bash"):
122
+ continue
123
+ cmd = tc.input_data.get("command", "")
124
+ if not cmd:
125
+ continue
126
+
127
+ if tc.is_error and tc.error_category == ErrorCategory.MODULE_NOT_FOUND:
128
+ prefix = _extract_python_command(cmd)
129
+ if prefix:
130
+ python_failures[prefix] += 1
131
+ python_sessions[prefix].add(session.session_id)
132
+ elif not tc.is_error:
133
+ prefix = _extract_python_command(cmd)
134
+ if prefix:
135
+ python_successes[prefix] += 1
136
+
137
+ if python_failures:
138
+ wrong = sorted(python_failures.keys(), key=lambda x: -python_failures[x])
139
+ correct = None
140
+ for cmd, _count in python_successes.most_common():
141
+ if cmd not in python_failures or python_successes[cmd] > python_failures[cmd] * 2:
142
+ correct = cmd
143
+ break
144
+ if correct and wrong:
145
+ total_evidence = sum(python_failures[w] for w in wrong)
146
+ total_sessions = len(set().union(*(python_sessions[w] for w in wrong)))
147
+ facts.append(
148
+ EnvironmentFact(
149
+ category="python",
150
+ correct_command=correct,
151
+ wrong_commands=wrong[:5],
152
+ evidence_count=total_evidence,
153
+ sessions_seen=total_sessions,
154
+ )
155
+ )
156
+
157
+ return facts
158
+
159
+
160
+ # =============================================================================
161
+ # Structure Analyzer (uses corrections to learn correct paths)
162
+ # =============================================================================
163
+
164
+
165
+ def _analyze_structure(
166
+ sessions: list[SessionData], corrections: list[Correction]
167
+ ) -> list[StructureNote]:
168
+ """Find file structure issues and learn correct paths from corrections."""
169
+ notes: list[StructureNote] = []
170
+
171
+ # 1. Path corrections: wrong path → correct path (from success correlation)
172
+ path_corrections: dict[str, Counter[str]] = defaultdict(Counter)
173
+ for c in corrections:
174
+ if c.tool_name not in ("Read", "read"):
175
+ continue
176
+ if c.error_category != ErrorCategory.FILE_NOT_FOUND:
177
+ continue
178
+ failed_path = c.failed_input.get("file_path", "")
179
+ success_path = c.success_input.get("file_path", "")
180
+ if failed_path and success_path and failed_path != success_path:
181
+ path_corrections[failed_path][success_path] += 1
182
+
183
+ for wrong_path, correct_paths in path_corrections.items():
184
+ best_correct, count = correct_paths.most_common(1)[0]
185
+ # Make paths relative to project for readability
186
+ wrong_short = _shorten_path(wrong_path)
187
+ correct_short = _shorten_path(best_correct)
188
+ notes.append(
189
+ StructureNote(
190
+ category="path_correction",
191
+ path=wrong_short,
192
+ correct_path=correct_short,
193
+ note=f"Not at `{wrong_short}` → actually at `{correct_short}`",
194
+ evidence_count=count,
195
+ )
196
+ )
197
+
198
+ # 2. Grep scope corrections: narrow path → broader path worked
199
+ scope_corrections: dict[str, Counter[str]] = defaultdict(Counter)
200
+ for c in corrections:
201
+ if c.tool_name not in ("Grep", "grep"):
202
+ continue
203
+ failed_path = c.failed_input.get("path", "")
204
+ success_path = c.success_input.get("path", "")
205
+ if failed_path and success_path and failed_path != success_path:
206
+ scope_corrections[_shorten_path(failed_path)][_shorten_path(success_path)] += 1
207
+
208
+ for wrong_scope, correct_scopes in scope_corrections.items():
209
+ best_scope, count = correct_scopes.most_common(1)[0]
210
+ notes.append(
211
+ StructureNote(
212
+ category="search_scope",
213
+ path=wrong_scope,
214
+ correct_path=best_scope,
215
+ note=f"Grep fails at `{wrong_scope}` → use `{best_scope}` instead",
216
+ evidence_count=count,
217
+ )
218
+ )
219
+
220
+ # 3. Large files (from raw failures, no correction needed)
221
+ large_files: Counter[str] = Counter()
222
+ large_sessions: dict[str, set[str]] = defaultdict(set)
223
+ for session in sessions:
224
+ for tc in session.tool_calls:
225
+ if (
226
+ tc.name in ("Read", "read")
227
+ and tc.is_error
228
+ and tc.error_category == ErrorCategory.FILE_TOO_LARGE
229
+ ):
230
+ path = tc.input_data.get("file_path", "")
231
+ if path:
232
+ short = _shorten_path(path)
233
+ large_files[short] += 1
234
+ large_sessions[short].add(session.session_id)
235
+
236
+ for path, count in large_files.most_common(10):
237
+ if count < 2:
238
+ break
239
+ notes.append(
240
+ StructureNote(
241
+ category="large_file",
242
+ path=path,
243
+ note=f"Too large for full read — always use offset/limit ({count} failures, {len(large_sessions[path])} sessions)",
244
+ evidence_count=count,
245
+ sessions_seen=len(large_sessions[path]),
246
+ )
247
+ )
248
+
249
+ # 4. Persistent missing paths (no correction found — file truly doesn't exist)
250
+ missing_no_correction: Counter[str] = Counter()
251
+ missing_sessions: dict[str, set[str]] = defaultdict(set)
252
+ corrected_paths = set(path_corrections.keys())
253
+ for session in sessions:
254
+ for tc in session.tool_calls:
255
+ if (
256
+ tc.name in ("Read", "read")
257
+ and tc.is_error
258
+ and tc.error_category == ErrorCategory.FILE_NOT_FOUND
259
+ ):
260
+ path = tc.input_data.get("file_path", "")
261
+ if path and path not in corrected_paths:
262
+ missing_no_correction[path] += 1
263
+ missing_sessions[path].add(session.session_id)
264
+
265
+ for path, count in missing_no_correction.most_common(10):
266
+ if count < 2:
267
+ break
268
+ short = _shorten_path(path)
269
+ notes.append(
270
+ StructureNote(
271
+ category="missing_path",
272
+ path=short,
273
+ note=f"Does not exist ({count} attempts, {len(missing_sessions[path])} sessions)",
274
+ evidence_count=count,
275
+ sessions_seen=len(missing_sessions[path]),
276
+ )
277
+ )
278
+
279
+ return notes
280
+
281
+
282
+ # =============================================================================
283
+ # Command Pattern Analyzer (uses corrections to learn command patterns)
284
+ # =============================================================================
285
+
286
+
287
+ def _analyze_commands(
288
+ sessions: list[SessionData], corrections: list[Correction]
289
+ ) -> list[CommandPattern]:
290
+ """Learn specific command patterns from Bash failure→success corrections."""
291
+ patterns: list[CommandPattern] = []
292
+
293
+ # Analyze Bash corrections
294
+ bash_corrections = [c for c in corrections if c.tool_name in ("Bash", "bash")]
295
+
296
+ # Group by error category to find patterns
297
+ by_category: dict[ErrorCategory, list[Correction]] = defaultdict(list)
298
+ for c in bash_corrections:
299
+ by_category[c.error_category].append(c)
300
+
301
+ # User-rejected commands: model should suggest, not execute
302
+ rejected = by_category.get(ErrorCategory.USER_REJECTED, [])
303
+ if rejected:
304
+ # Find the most commonly rejected command patterns
305
+ rejected_cmds: Counter[str] = Counter()
306
+ for c in rejected:
307
+ cmd = c.failed_input.get("command", "")
308
+ base = _extract_command_signature(cmd)
309
+ if base:
310
+ rejected_cmds[base] += 1
311
+
312
+ for cmd_sig, count in rejected_cmds.most_common(5):
313
+ if count < 2:
314
+ break
315
+ patterns.append(
316
+ CommandPattern(
317
+ category="user_prefers_manual",
318
+ wrong_pattern=f"Executing: {cmd_sig}",
319
+ correct_pattern="Show the command to the user and let them run it",
320
+ explanation=f"User rejected this command {count} times — they prefer to run it themselves",
321
+ evidence_count=count,
322
+ sessions_seen=len(
323
+ {
324
+ c.session_id
325
+ for c in rejected
326
+ if _extract_command_signature(c.failed_input.get("command", ""))
327
+ == cmd_sig
328
+ }
329
+ ),
330
+ )
331
+ )
332
+
333
+ # Build failures: learn what command form works
334
+ build_fails = by_category.get(ErrorCategory.BUILD_FAILURE, [])
335
+ for c in build_fails:
336
+ failed_cmd = c.failed_input.get("command", "")
337
+ success_cmd = c.success_input.get("command", "")
338
+ if failed_cmd and success_cmd:
339
+ patterns.append(
340
+ CommandPattern(
341
+ category="build",
342
+ wrong_pattern=_extract_command_signature(failed_cmd),
343
+ correct_pattern=_extract_command_signature(success_cmd),
344
+ explanation="Build failed with first form, succeeded with second",
345
+ evidence_count=1,
346
+ )
347
+ )
348
+
349
+ # Module not found: learn correct python invocation
350
+ module_fails = by_category.get(ErrorCategory.MODULE_NOT_FOUND, [])
351
+ if module_fails:
352
+ wrong_pythons: Counter[str] = Counter()
353
+ correct_pythons: Counter[str] = Counter()
354
+ for c in module_fails:
355
+ wp = _extract_python_command(c.failed_input.get("command", ""))
356
+ cp = _extract_python_command(c.success_input.get("command", ""))
357
+ if wp:
358
+ wrong_pythons[wp] += 1
359
+ if cp:
360
+ correct_pythons[cp] += 1
361
+
362
+ if wrong_pythons and correct_pythons:
363
+ wrong = wrong_pythons.most_common(1)[0][0]
364
+ correct = correct_pythons.most_common(1)[0][0]
365
+ if wrong != correct:
366
+ patterns.append(
367
+ CommandPattern(
368
+ category="python_runtime",
369
+ wrong_pattern=f"`{wrong}` (modules not available)",
370
+ correct_pattern=f"`{correct}` (has project dependencies)",
371
+ explanation=f"Using `{wrong}` causes ModuleNotFoundError — use `{correct}` which has the project's venv",
372
+ evidence_count=sum(wrong_pythons.values()),
373
+ )
374
+ )
375
+
376
+ # Deduplicate patterns
377
+ seen = set()
378
+ unique = []
379
+ for p in patterns:
380
+ key = (p.category, p.wrong_pattern[:50])
381
+ if key not in seen:
382
+ seen.add(key)
383
+ unique.append(p)
384
+ return unique
385
+
386
+
387
+ # =============================================================================
388
+ # Retry Analyzer (uses corrections to provide specific suggestions)
389
+ # =============================================================================
390
+
391
+
392
+ def _analyze_retries(
393
+ sessions: list[SessionData], corrections: list[Correction]
394
+ ) -> list[RetryPattern]:
395
+ """Find stubborn retries with specific fix suggestions from corrections."""
396
+ patterns: list[RetryPattern] = []
397
+
398
+ # Build a correction lookup: (tool, error_category) → list of corrections
399
+ correction_lookup: dict[tuple[str, str], list[Correction]] = defaultdict(list)
400
+ for c in corrections:
401
+ correction_lookup[(c.tool_name, c.error_category.value)].append(c)
402
+
403
+ # Find retry streaks
404
+ pattern_counter: Counter[tuple[str, str, str]] = Counter()
405
+ max_retries: dict[tuple[str, str, str], int] = {}
406
+
407
+ for session in sessions:
408
+ streak: dict[str, list[ToolCall]] = defaultdict(list)
409
+ for tc in session.tool_calls:
410
+ key = f"{tc.name}:{tc.error_category.value}"
411
+ if tc.is_error:
412
+ streak[key].append(tc)
413
+ else:
414
+ if len(streak.get(key, [])) >= 3:
415
+ calls = streak[key]
416
+ pk = (tc.name, calls[0].error_category.value, calls[0].input_summary[:50])
417
+ pattern_counter[pk] += 1
418
+ max_retries[pk] = max(max_retries.get(pk, 0), len(calls))
419
+ streak[key] = []
420
+ for _key, calls in streak.items():
421
+ if len(calls) >= 3:
422
+ pk = (calls[0].name, calls[0].error_category.value, calls[0].input_summary[:50])
423
+ pattern_counter[pk] += 1
424
+ max_retries[pk] = max(max_retries.get(pk, 0), len(calls))
425
+
426
+ for (tool, err_cat, input_key), count in pattern_counter.most_common(10):
427
+ max_r = max_retries.get((tool, err_cat, input_key), 3)
428
+
429
+ # Try to get a SPECIFIC suggestion from corrections
430
+ relevant_corrections = correction_lookup.get((tool, err_cat), [])
431
+ suggestion = _build_specific_suggestion(tool, err_cat, relevant_corrections)
432
+
433
+ patterns.append(
434
+ RetryPattern(
435
+ tool_name=tool,
436
+ error_category=ErrorCategory(err_cat),
437
+ description=f"{tool} failing with {err_cat}: {input_key}",
438
+ max_retries_seen=max_r,
439
+ suggestion=suggestion,
440
+ evidence_count=count,
441
+ )
442
+ )
443
+
444
+ return patterns
445
+
446
+
447
+ def _build_specific_suggestion(
448
+ tool: str, error_category: str, corrections: list[Correction]
449
+ ) -> str:
450
+ """Build a specific suggestion from actual corrections, not generic advice."""
451
+ if not corrections:
452
+ # No corrections available — use tool+error specific defaults
453
+ return _default_suggestion(tool, error_category)
454
+
455
+ # Summarize what corrections tell us
456
+ if tool in ("Read", "read") and error_category == "file_not_found":
457
+ examples = []
458
+ for c in corrections[:3]:
459
+ wrong = _shorten_path(c.failed_input.get("file_path", ""))
460
+ right = _shorten_path(c.success_input.get("file_path", ""))
461
+ if wrong and right:
462
+ examples.append(f"`{wrong}` → `{right}`")
463
+ if examples:
464
+ return "Use Glob to discover actual path. Known corrections: " + "; ".join(examples)
465
+
466
+ if tool in ("Grep", "grep"):
467
+ # Summarize scope corrections
468
+ scopes = set()
469
+ for c in corrections[:5]:
470
+ right_path = c.success_input.get("path", "")
471
+ if right_path:
472
+ scopes.add(_shorten_path(right_path))
473
+ if scopes:
474
+ return f"Scope searches to: {', '.join(sorted(scopes)[:3])}"
475
+
476
+ if tool in ("Bash", "bash") and error_category == "user_rejected":
477
+ return "User prefers to run this command themselves. Show the command, don't execute it."
478
+
479
+ if tool in ("Bash", "bash") and error_category == "module_not_found":
480
+ correct_cmds = set()
481
+ for c in corrections[:5]:
482
+ prefix = _extract_python_command(c.success_input.get("command", ""))
483
+ if prefix:
484
+ correct_cmds.add(prefix)
485
+ if correct_cmds:
486
+ return f"Use {' or '.join(sorted(correct_cmds))} (has project dependencies)"
487
+
488
+ # Fallback: show one correction example
489
+ c = corrections[0]
490
+ return f"What worked: {c.success_summary[:80]}"
491
+
492
+
493
+ def _default_suggestion(tool: str, error_category: str) -> str:
494
+ """Fallback when no corrections are available."""
495
+ defaults = {
496
+ (
497
+ "Glob",
498
+ "no_matches",
499
+ ): "Broaden pattern to **/*.ext or use ls to explore directory structure",
500
+ ("Grep", "no_matches"): "Try case-insensitive (-i) or broaden search scope",
501
+ ("Grep", "timeout"): "Scope Grep to a specific subdirectory — the full repo is too large",
502
+ ("Read", "file_not_found"): "Use Glob to discover the file path before Read",
503
+ ("Read", "file_too_large"): "Use offset/limit parameters for this file",
504
+ ("Bash", "module_not_found"): "Use the project's virtualenv Python",
505
+ ("Bash", "permission_denied"): "Do not retry — try a different approach",
506
+ ("Bash", "command_not_found"): "Verify tool is installed: which <tool>",
507
+ ("Bash", "user_rejected"): "User does not want this command executed. Show it instead.",
508
+ ("Edit", "unknown"): "If old_string has multiple matches, add more surrounding context",
509
+ }
510
+ return defaults.get((tool, error_category), "Try an alternative approach after 2 failures")
511
+
512
+
513
+ # =============================================================================
514
+ # Permission Analyzer
515
+ # =============================================================================
516
+
517
+
518
+ def _analyze_permissions(sessions: list[SessionData]) -> list[str]:
519
+ """Find commands repeatedly denied — with specific advice."""
520
+ denied: Counter[str] = Counter()
521
+ denied_cmds: dict[str, str] = {} # key → full command example
522
+
523
+ for session in sessions:
524
+ for tc in session.tool_calls:
525
+ if not tc.is_error:
526
+ continue
527
+ if tc.error_category not in (
528
+ ErrorCategory.PERMISSION_DENIED,
529
+ ErrorCategory.USER_REJECTED,
530
+ ):
531
+ continue
532
+ sig = _extract_command_signature(
533
+ tc.input_data.get("command", "")
534
+ if tc.name in ("Bash", "bash")
535
+ else tc.input_summary
536
+ )
537
+ key = f"{tc.name}: {sig}"
538
+ denied[key] += 1
539
+ if key not in denied_cmds:
540
+ denied_cmds[key] = tc.input_summary[:80]
541
+
542
+ results = []
543
+ for key, count in denied.most_common(10):
544
+ if count < 3:
545
+ break
546
+ results.append(
547
+ f"{key} — denied {count} times. Show the command to the user instead of executing it."
548
+ )
549
+ return results
550
+
551
+
552
+ # =============================================================================
553
+ # Cross-Session Analyzer
554
+ # =============================================================================
555
+
556
+
557
+ def _analyze_cross_session(sessions: list[SessionData]) -> list[str]:
558
+ """Find failure patterns that repeat across 3+ sessions."""
559
+ pattern_sessions: dict[str, set[str]] = defaultdict(set)
560
+
561
+ for session in sessions:
562
+ for tc in session.tool_calls:
563
+ if not tc.is_error or tc.error_category == ErrorCategory.SIBLING_ERROR:
564
+ continue
565
+ key = f"{tc.name}|{tc.error_category.value}|{tc.input_summary[:60]}"
566
+ pattern_sessions[key].add(session.session_id)
567
+
568
+ cross_session = []
569
+ for key, session_ids in sorted(pattern_sessions.items(), key=lambda x: -len(x[1])):
570
+ if len(session_ids) < 3:
571
+ continue
572
+ parts = key.split("|", 2)
573
+ tool, err, inp = parts[0], parts[1], parts[2] if len(parts) > 2 else "?"
574
+ cross_session.append(f"{tool} {err}: {inp} (across {len(session_ids)} sessions)")
575
+ if len(cross_session) >= 15:
576
+ break
577
+
578
+ return cross_session
579
+
580
+
581
+ # =============================================================================
582
+ # Helpers
583
+ # =============================================================================
584
+
585
+
586
+ _PYTHON_CMD_RE = re.compile(
587
+ r"^((?:source\s+\S+\s*&&\s*)?(?:\.venv/bin/)?(?:python3?|uv run python|uv run|/opt/nflx/python))"
588
+ )
589
+
590
+
591
+ def _extract_python_command(cmd: str) -> str | None:
592
+ """Extract the python invocation prefix from a command."""
593
+ cmd = cmd.strip()
594
+ if "&&" in cmd:
595
+ for part in cmd.split("&&"):
596
+ result = _extract_python_command(part.strip())
597
+ if result:
598
+ return result
599
+ return None
600
+ m = _PYTHON_CMD_RE.match(cmd)
601
+ return m.group(1) if m else None
602
+
603
+
604
+ def _extract_command_signature(cmd: str) -> str:
605
+ """Extract a normalizable command signature (first ~60 chars, no args)."""
606
+ cmd = cmd.strip()
607
+ # Truncate at first newline
608
+ if "\n" in cmd:
609
+ cmd = cmd.split("\n")[0]
610
+ return cmd[:60]
611
+
612
+
613
+ def _shorten_path(path: str) -> str:
614
+ """Make a path relative to home for readability."""
615
+ home = os.path.expanduser("~")
616
+ if path.startswith(home):
617
+ return "~" + path[len(home) :]
618
+ return path
headroom/learn/models.py ADDED
@@ -0,0 +1,240 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Data models for Headroom Learn — tool-agnostic abstractions.
2
+
3
+ These models normalize tool call data from ANY agent system (Claude Code, Cursor,
4
+ Codex, custom agents) into a common format that analyzers can work with.
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ from dataclasses import dataclass, field
10
+ from datetime import datetime
11
+ from enum import Enum
12
+ from pathlib import Path
13
+
14
+ # =============================================================================
15
+ # Error Classification
16
+ # =============================================================================
17
+
18
+
19
+ class ErrorCategory(str, Enum):
20
+ """Classified error categories for tool call failures."""
21
+
22
+ FILE_NOT_FOUND = "file_not_found"
23
+ MODULE_NOT_FOUND = "module_not_found"
24
+ COMMAND_NOT_FOUND = "command_not_found"
25
+ PERMISSION_DENIED = "permission_denied"
26
+ FILE_TOO_LARGE = "file_too_large"
27
+ IS_DIRECTORY = "is_directory"
28
+ SYNTAX_ERROR = "syntax_error"
29
+ RUNTIME_ERROR = "runtime_error"
30
+ TIMEOUT = "timeout"
31
+ NO_MATCHES = "no_matches" # Grep/Glob found nothing
32
+ USER_REJECTED = "user_rejected"
33
+ SIBLING_ERROR = "sibling_error" # Cascade from parallel call failure
34
+ EXIT_CODE = "exit_code"
35
+ CONNECTION_ERROR = "connection_error"
36
+ BUILD_FAILURE = "build_failure"
37
+ UNKNOWN = "unknown"
38
+
39
+
40
+ # =============================================================================
41
+ # Core Data Models (Tool-Agnostic)
42
+ # =============================================================================
43
+
44
+
45
+ @dataclass
46
+ class ToolCall:
47
+ """A single tool call and its result — normalized from any agent system.
48
+
49
+ This is the fundamental unit of analysis. Scanners produce these,
50
+ analyzers consume them.
51
+ """
52
+
53
+ name: str # Tool name ("Bash", "Read", "file_search", etc.)
54
+ tool_call_id: str # Unique ID linking call to result
55
+ input_data: dict # Tool input parameters
56
+ output: str # Result content (may be error message)
57
+ is_error: bool # Whether the call failed
58
+ error_category: ErrorCategory = ErrorCategory.UNKNOWN
59
+ msg_index: int = 0 # Position in conversation
60
+ output_bytes: int = 0 # Size of output
61
+
62
+ @property
63
+ def input_summary(self) -> str:
64
+ """Short summary of tool input for display."""
65
+ if self.name in ("Bash", "bash"):
66
+ cmd: str = self.input_data.get("command", "")
67
+ return cmd[:100] + "..." if len(cmd) > 100 else cmd
68
+ if self.name in ("Read", "read"):
69
+ return str(self.input_data.get("file_path", "?"))
70
+ if self.name in ("Grep", "grep"):
71
+ return str(self.input_data.get("pattern", "?"))
72
+ if self.name in ("Glob", "glob"):
73
+ return str(self.input_data.get("pattern", "?"))
74
+ if self.name in ("Edit", "edit", "Write", "write"):
75
+ return str(self.input_data.get("file_path", "?"))
76
+ return str(self.input_data)[:80]
77
+
78
+
79
+ @dataclass
80
+ class SessionData:
81
+ """Normalized data from a single conversation session."""
82
+
83
+ session_id: str
84
+ tool_calls: list[ToolCall] = field(default_factory=list)
85
+ timestamp: datetime | None = None
86
+
87
+ @property
88
+ def failure_count(self) -> int:
89
+ return sum(1 for tc in self.tool_calls if tc.is_error)
90
+
91
+ @property
92
+ def failure_rate(self) -> float:
93
+ if not self.tool_calls:
94
+ return 0.0
95
+ return self.failure_count / len(self.tool_calls)
96
+
97
+
98
+ @dataclass
99
+ class ProjectInfo:
100
+ """Information about a project discovered by a scanner."""
101
+
102
+ name: str # Human-readable project name
103
+ project_path: Path # Actual project directory
104
+ data_path: Path # Where conversation logs are stored
105
+ context_file: Path | None = None # CLAUDE.md / .cursorrules / AGENTS.md
106
+ memory_file: Path | None = None # MEMORY.md or equivalent
107
+
108
+
109
+ # =============================================================================
110
+ # Analysis Output Models
111
+ # =============================================================================
112
+
113
+
114
+ class RecommendationTarget(str, Enum):
115
+ """Where a recommendation should be written."""
116
+
117
+ CONTEXT_FILE = "context_file" # CLAUDE.md, .cursorrules, AGENTS.md
118
+ MEMORY_FILE = "memory_file" # MEMORY.md or equivalent
119
+
120
+
121
+ @dataclass
122
+ class EnvironmentFact:
123
+ """A learned fact about the project's runtime environment."""
124
+
125
+ category: str # "python", "build_tool", "test_runner", "linter"
126
+ correct_command: str # What works: "uv run python"
127
+ wrong_commands: list[str] = field(default_factory=list) # What fails: ["python3"]
128
+ evidence_count: int = 0 # How many failures support this
129
+ sessions_seen: int = 0 # Across how many sessions
130
+
131
+
132
+ @dataclass
133
+ class StructureNote:
134
+ """A learned fact about the project's file structure."""
135
+
136
+ category: str # "large_file", "missing_path", "path_correction", "search_scope"
137
+ path: str # The file path in question
138
+ note: str # Human-readable note
139
+ correct_path: str = "" # If corrected, what the actual path is
140
+ evidence_count: int = 0
141
+ sessions_seen: int = 0
142
+
143
+
144
+ @dataclass
145
+ class Correction:
146
+ """A failure→success pair: what failed and what worked instead.
147
+
148
+ This is the core learning primitive. By comparing the failed input to
149
+ the successful input, we extract specific actionable knowledge.
150
+ """
151
+
152
+ tool_name: str
153
+ failed_input: dict # The input that failed
154
+ success_input: dict # The input that succeeded
155
+ error_category: ErrorCategory
156
+ session_id: str
157
+
158
+ @property
159
+ def failed_summary(self) -> str:
160
+ if self.tool_name in ("Read", "read"):
161
+ return str(self.failed_input.get("file_path", "?"))
162
+ if self.tool_name in ("Bash", "bash"):
163
+ return str(self.failed_input.get("command", "?"))[:100]
164
+ if self.tool_name in ("Grep", "grep"):
165
+ path = str(self.failed_input.get("path", ""))
166
+ pattern = str(self.failed_input.get("pattern", ""))
167
+ return f"pattern={pattern[:40]} path={path}"
168
+ return str(self.failed_input)[:80]
169
+
170
+ @property
171
+ def success_summary(self) -> str:
172
+ if self.tool_name in ("Read", "read"):
173
+ return str(self.success_input.get("file_path", "?"))
174
+ if self.tool_name in ("Bash", "bash"):
175
+ return str(self.success_input.get("command", "?"))[:100]
176
+ if self.tool_name in ("Grep", "grep"):
177
+ path = str(self.success_input.get("path", ""))
178
+ pattern = str(self.success_input.get("pattern", ""))
179
+ return f"pattern={pattern[:40]} path={path}"
180
+ return str(self.success_input)[:80]
181
+
182
+
183
+ @dataclass
184
+ class CommandPattern:
185
+ """A learned pattern about how commands should be run in this project."""
186
+
187
+ category: str # "gradle", "python", "test", "build", "lint"
188
+ wrong_pattern: str # What fails (e.g., "cd /path && ./gradlew")
189
+ correct_pattern: str # What works (e.g., "../gradlew from axion/")
190
+ explanation: str # Why (e.g., "user rejects cd-based gradle, use relative path")
191
+ evidence_count: int = 0
192
+ sessions_seen: int = 0
193
+
194
+
195
+ @dataclass
196
+ class RetryPattern:
197
+ """A pattern of stubborn retries that should be prevented."""
198
+
199
+ tool_name: str
200
+ error_category: ErrorCategory
201
+ description: str # What keeps failing
202
+ max_retries_seen: int # Worst case observed
203
+ suggestion: str # What to do instead (SPECIFIC, from success correlation)
204
+ evidence_count: int = 0
205
+
206
+
207
+ @dataclass
208
+ class Recommendation:
209
+ """A concrete recommendation to write to a context/memory file."""
210
+
211
+ target: RecommendationTarget
212
+ section: str # Section heading (e.g., "Environment", "Known Large Files")
213
+ content: str # Markdown content for the section
214
+ confidence: float = 0.0 # 0-1, based on evidence strength
215
+ evidence_count: int = 0 # Number of failures supporting this
216
+
217
+
218
+ @dataclass
219
+ class AnalysisReport:
220
+ """Complete output of failure analysis for a project."""
221
+
222
+ project: ProjectInfo
223
+ total_calls: int = 0
224
+ total_failures: int = 0
225
+ total_sessions: int = 0
226
+ waste_bytes: int = 0
227
+
228
+ environment_facts: list[EnvironmentFact] = field(default_factory=list)
229
+ structure_notes: list[StructureNote] = field(default_factory=list)
230
+ retry_patterns: list[RetryPattern] = field(default_factory=list)
231
+ command_patterns: list[CommandPattern] = field(default_factory=list)
232
+ corrections: list[Correction] = field(default_factory=list)
233
+ permission_issues: list[str] = field(default_factory=list)
234
+ cross_session_patterns: list[str] = field(default_factory=list)
235
+
236
+ @property
237
+ def failure_rate(self) -> float:
238
+ if not self.total_calls:
239
+ return 0.0
240
+ return self.total_failures / self.total_calls
headroom/learn/scanner.py ADDED
@@ -0,0 +1,358 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Conversation scanners — read tool call logs from different agent systems.
2
+
3
+ Scanners normalize conversation data into ToolCall sequences that analyzers
4
+ can process regardless of the source system.
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ import json
10
+ import logging
11
+ import re
12
+ from abc import ABC, abstractmethod
13
+ from pathlib import Path
14
+
15
+ from .models import (
16
+ ErrorCategory,
17
+ ProjectInfo,
18
+ SessionData,
19
+ ToolCall,
20
+ )
21
+
22
+ logger = logging.getLogger(__name__)
23
+
24
+ # =============================================================================
25
+ # Error Classification
26
+ # =============================================================================
27
+
28
+ # Patterns checked in order — first match wins
29
+ _ERROR_PATTERNS: list[tuple[re.Pattern, ErrorCategory]] = [
30
+ (
31
+ re.compile(r"No such file or directory|ENOENT|FileNotFoundError|does not exist", re.I),
32
+ ErrorCategory.FILE_NOT_FOUND,
33
+ ),
34
+ (
35
+ re.compile(r"ModuleNotFoundError|ImportError|No module named", re.I),
36
+ ErrorCategory.MODULE_NOT_FOUND,
37
+ ),
38
+ (re.compile(r"command not found", re.I), ErrorCategory.COMMAND_NOT_FOUND),
39
+ (
40
+ re.compile(r"Permission denied|EACCES|EPERM|auto-denied", re.I),
41
+ ErrorCategory.PERMISSION_DENIED,
42
+ ),
43
+ (
44
+ re.compile(r"file is too large|too many lines|exceeds.*limit", re.I),
45
+ ErrorCategory.FILE_TOO_LARGE,
46
+ ),
47
+ (re.compile(r"EISDIR|Is a directory", re.I), ErrorCategory.IS_DIRECTORY),
48
+ (re.compile(r"SyntaxError|IndentationError", re.I), ErrorCategory.SYNTAX_ERROR),
49
+ (re.compile(r"Traceback \(most recent|Exception:|Error:", re.I), ErrorCategory.RUNTIME_ERROR),
50
+ (re.compile(r"timed? ?out|TimeoutError|deadline exceeded", re.I), ErrorCategory.TIMEOUT),
51
+ (re.compile(r"No (?:matches|files|results) found|0 matches", re.I), ErrorCategory.NO_MATCHES),
52
+ (
53
+ re.compile(r"user.*reject|user.*denied|declined|didn't want to proceed", re.I),
54
+ ErrorCategory.USER_REJECTED,
55
+ ),
56
+ (re.compile(r"[Ss]ibling tool call errored", re.I), ErrorCategory.SIBLING_ERROR),
57
+ (re.compile(r"exit code|non-zero|exited with", re.I), ErrorCategory.EXIT_CODE),
58
+ (
59
+ re.compile(r"ConnectionError|ConnectionRefused|ECONNREFUSED|network", re.I),
60
+ ErrorCategory.CONNECTION_ERROR,
61
+ ),
62
+ (
63
+ re.compile(r"BUILD FAILED|compilation error|compile error", re.I),
64
+ ErrorCategory.BUILD_FAILURE,
65
+ ),
66
+ ]
67
+
68
+
69
+ def classify_error(content: str) -> ErrorCategory:
70
+ """Classify an error message into a category."""
71
+ for pattern, category in _ERROR_PATTERNS:
72
+ if pattern.search(content[:2000]): # Only check first 2KB
73
+ return category
74
+ return ErrorCategory.UNKNOWN
75
+
76
+
77
+ def is_error_content(content: str) -> bool:
78
+ """Heuristic: does this tool result look like an error?"""
79
+ if not content or len(content) < 10:
80
+ return False
81
+ # Check for common error indicators in first 1KB
82
+ snippet = content[:1000]
83
+ indicators = [
84
+ "Error:",
85
+ "error:",
86
+ "ENOENT",
87
+ "No such file",
88
+ "command not found",
89
+ "Permission denied",
90
+ "ModuleNotFoundError",
91
+ "Traceback (most recent",
92
+ "FAILED",
93
+ "EISDIR",
94
+ "auto-denied",
95
+ "Sibling tool call errored",
96
+ "timed out",
97
+ "exit code",
98
+ "FileNotFoundError",
99
+ ]
100
+ return any(ind in snippet for ind in indicators)
101
+
102
+
103
+ # =============================================================================
104
+ # Abstract Scanner
105
+ # =============================================================================
106
+
107
+
108
+ class ConversationScanner(ABC):
109
+ """Base class for scanning conversation logs from any agent system.
110
+
111
+ Subclasses implement log format parsing for specific tools (Claude Code,
112
+ Cursor, Codex, etc.) and produce normalized ToolCall sequences.
113
+ """
114
+
115
+ @abstractmethod
116
+ def discover_projects(self) -> list[ProjectInfo]:
117
+ """Discover all projects with conversation data."""
118
+ ...
119
+
120
+ @abstractmethod
121
+ def scan_project(self, project: ProjectInfo) -> list[SessionData]:
122
+ """Scan all sessions for a project, returning normalized tool calls."""
123
+ ...
124
+
125
+
126
+ # =============================================================================
127
+ # Claude Code Scanner
128
+ # =============================================================================
129
+
130
+
131
+ class ClaudeCodeScanner(ConversationScanner):
132
+ """Reads Claude Code conversation logs from ~/.claude/projects/.
133
+
134
+ Claude Code stores conversations as JSONL files with these line types:
135
+ - type="assistant": message.content[] has tool_use blocks (name, input, id)
136
+ - type="user": message.content[] has tool_result blocks (tool_use_id, content)
137
+ """
138
+
139
+ def __init__(self, claude_dir: Path | None = None):
140
+ self.claude_dir = claude_dir or Path.home() / ".claude"
141
+ self.projects_dir = self.claude_dir / "projects"
142
+
143
+ def discover_projects(self) -> list[ProjectInfo]:
144
+ """Discover all projects under ~/.claude/projects/."""
145
+ if not self.projects_dir.exists():
146
+ return []
147
+
148
+ projects = []
149
+ for entry in sorted(self.projects_dir.iterdir()):
150
+ if not entry.is_dir() or entry.name.startswith("."):
151
+ continue
152
+
153
+ # Decode project path from escaped directory name
154
+ # e.g., "-Users-tchopra-claude-projects-headroom" → "/Users/tchopra/claude-projects/headroom"
155
+ project_path = Path("/" + entry.name.replace("-", "/", entry.name.count("-")))
156
+
157
+ # Try smarter decoding: split on segments that look like path components
158
+ # The escaping replaces / with - but also - in names stays as -
159
+ # Heuristic: try the decoded path, if it exists use it
160
+ decoded = _decode_project_path(entry.name)
161
+ if decoded:
162
+ project_path = decoded
163
+
164
+ # Derive human-readable name
165
+ name = project_path.name if project_path != Path("/") else entry.name
166
+
167
+ # Check for CLAUDE.md in actual project directory
168
+ context_file = None
169
+ if project_path.exists():
170
+ claude_md = project_path / "CLAUDE.md"
171
+ if claude_md.exists():
172
+ context_file = claude_md
173
+
174
+ # Check for MEMORY.md
175
+ memory_dir = entry / "memory"
176
+ memory_file = memory_dir / "MEMORY.md" if memory_dir.exists() else None
177
+ if memory_file and not memory_file.exists():
178
+ memory_file = None
179
+
180
+ # Only include projects with JSONL files
181
+ jsonl_files = list(entry.glob("*.jsonl"))
182
+ if not jsonl_files:
183
+ continue
184
+
185
+ projects.append(
186
+ ProjectInfo(
187
+ name=name,
188
+ project_path=project_path,
189
+ data_path=entry,
190
+ context_file=context_file,
191
+ memory_file=memory_file,
192
+ )
193
+ )
194
+
195
+ return projects
196
+
197
+ def scan_project(self, project: ProjectInfo) -> list[SessionData]:
198
+ """Scan all conversation JSONL files for a project."""
199
+ sessions = []
200
+
201
+ # Find all JSONL files (main conversations, not subagent files)
202
+ jsonl_files = sorted(project.data_path.glob("*.jsonl"))
203
+
204
+ for jsonl_path in jsonl_files:
205
+ session = self._scan_session(jsonl_path)
206
+ if session and session.tool_calls:
207
+ sessions.append(session)
208
+
209
+ return sessions
210
+
211
+ def _scan_session(self, jsonl_path: Path) -> SessionData | None:
212
+ """Scan a single JSONL conversation file."""
213
+ session_id = jsonl_path.stem
214
+ tool_uses: dict[str, tuple[str, dict]] = {} # tc_id → (tool_name, input)
215
+ tool_calls: list[ToolCall] = []
216
+ msg_index = 0
217
+
218
+ try:
219
+ with open(jsonl_path) as f:
220
+ for line in f:
221
+ try:
222
+ d = json.loads(line)
223
+ except json.JSONDecodeError:
224
+ continue
225
+
226
+ msg_index += 1
227
+ line_type = d.get("type", "")
228
+
229
+ if line_type == "assistant":
230
+ self._extract_tool_uses(d, tool_uses)
231
+ elif line_type == "user":
232
+ self._extract_tool_results(d, tool_uses, tool_calls, msg_index)
233
+
234
+ except (OSError, UnicodeDecodeError) as e:
235
+ logger.debug("Failed to read %s: %s", jsonl_path, e)
236
+ return None
237
+
238
+ return SessionData(session_id=session_id, tool_calls=tool_calls)
239
+
240
+ def _extract_tool_uses(self, d: dict, tool_uses: dict[str, tuple[str, dict]]) -> None:
241
+ """Extract tool_use blocks from an assistant message."""
242
+ msg = d.get("message", {})
243
+ content = msg.get("content", [])
244
+ if not isinstance(content, list):
245
+ return
246
+
247
+ for block in content:
248
+ if not isinstance(block, dict) or block.get("type") != "tool_use":
249
+ continue
250
+ tc_id = block.get("id", "")
251
+ name = block.get("name", "")
252
+ inp = block.get("input", {})
253
+ if tc_id and name:
254
+ tool_uses[tc_id] = (name, inp if isinstance(inp, dict) else {})
255
+
256
+ def _extract_tool_results(
257
+ self,
258
+ d: dict,
259
+ tool_uses: dict[str, tuple[str, dict]],
260
+ tool_calls: list[ToolCall],
261
+ msg_index: int,
262
+ ) -> None:
263
+ """Extract tool_result blocks from a user message and match to tool_uses."""
264
+ msg = d.get("message", {})
265
+ content = msg.get("content", [])
266
+ if not isinstance(content, list):
267
+ return
268
+
269
+ for block in content:
270
+ if not isinstance(block, dict) or block.get("type") != "tool_result":
271
+ continue
272
+
273
+ tc_id = block.get("tool_use_id", "")
274
+ result_content = block.get("content", "")
275
+ if not isinstance(result_content, str):
276
+ result_content = str(result_content)
277
+
278
+ # Match to tool_use
279
+ if tc_id not in tool_uses:
280
+ continue
281
+
282
+ name, inp = tool_uses[tc_id]
283
+
284
+ # Determine if error
285
+ explicit_error = block.get("is_error", False)
286
+ detected_error = is_error_content(result_content)
287
+ is_err = explicit_error or detected_error
288
+
289
+ error_cat = classify_error(result_content) if is_err else ErrorCategory.UNKNOWN
290
+
291
+ tool_calls.append(
292
+ ToolCall(
293
+ name=name,
294
+ tool_call_id=tc_id,
295
+ input_data=inp,
296
+ output=result_content,
297
+ is_error=is_err,
298
+ error_category=error_cat,
299
+ msg_index=msg_index,
300
+ output_bytes=len(result_content.encode("utf-8")),
301
+ )
302
+ )
303
+
304
+
305
+ def _decode_project_path(escaped_name: str) -> Path | None:
306
+ """Decode a Claude Code escaped project path.
307
+
308
+ Claude Code escapes paths by replacing / with -.
309
+ e.g., "-Users-tchopra-claude-projects-headroom"
310
+ → "/Users/tchopra/claude-projects/headroom"
311
+
312
+ Since - is ambiguous (path separator vs literal hyphen), we try
313
+ progressively and check which decoded path actually exists.
314
+ """
315
+ if not escaped_name.startswith("-"):
316
+ return None
317
+
318
+ # Simple approach: replace all - with / and check if path exists
319
+ simple = Path("/" + escaped_name[1:].replace("-", "/"))
320
+ if simple.exists():
321
+ return simple
322
+
323
+ # Try common patterns: /Users/username/...
324
+ parts = escaped_name[1:].split("-")
325
+ if len(parts) < 3:
326
+ return None
327
+
328
+ # Build path greedily: try joining with / and check existence
329
+ # Start with /Users/username (first 2 components are almost always correct)
330
+ if parts[0] == "Users" and len(parts) > 2:
331
+ base = Path(f"/{parts[0]}/{parts[1]}")
332
+ remaining = parts[2:]
333
+ return _greedy_path_decode(base, remaining)
334
+
335
+ return None
336
+
337
+
338
+ def _greedy_path_decode(base: Path, parts: list[str]) -> Path | None:
339
+ """Greedily decode remaining path parts, trying - as / first."""
340
+ if not parts:
341
+ return base if base.exists() else None
342
+
343
+ # Try using / (this part is a directory component)
344
+ slash_path = base / parts[0]
345
+ result = _greedy_path_decode(slash_path, parts[1:])
346
+ if result:
347
+ return result
348
+
349
+ # Try joining with - (this part has a literal hyphen)
350
+ if len(parts) > 1:
351
+ hyphen_name = f"{parts[0]}-{parts[1]}"
352
+ hyphen_path = base / hyphen_name
353
+ result = _greedy_path_decode(hyphen_path, parts[2:])
354
+ if result:
355
+ return result
356
+
357
+ # If we've exhausted parts, check if current path exists
358
+ return base if base.exists() else None
headroom/learn/writer.py ADDED
@@ -0,0 +1,333 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Context writers — write learned patterns to agent-specific context files.
2
+
3
+ Writers take Recommendations and write them to the appropriate context
4
+ injection mechanism for each agent system (CLAUDE.md, .cursorrules, etc.).
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ import re
10
+ from abc import ABC, abstractmethod
11
+ from datetime import datetime, timezone
12
+ from pathlib import Path
13
+
14
+ from .models import (
15
+ AnalysisReport,
16
+ CommandPattern,
17
+ EnvironmentFact,
18
+ ProjectInfo,
19
+ Recommendation,
20
+ RecommendationTarget,
21
+ RetryPattern,
22
+ StructureNote,
23
+ )
24
+
25
+ # Marker delimiters for Headroom-managed sections
26
+ _MARKER_START = "<!-- headroom:learn:start -->"
27
+ _MARKER_END = "<!-- headroom:learn:end -->"
28
+ _MARKER_PATTERN = re.compile(
29
+ re.escape(_MARKER_START) + r".*?" + re.escape(_MARKER_END),
30
+ re.DOTALL,
31
+ )
32
+
33
+
34
+ # =============================================================================
35
+ # Recommender: AnalysisReport → Recommendations
36
+ # =============================================================================
37
+
38
+
39
+ class Recommender:
40
+ """Converts an AnalysisReport into concrete markdown recommendations."""
41
+
42
+ def recommend(self, report: AnalysisReport) -> list[Recommendation]:
43
+ recommendations: list[Recommendation] = []
44
+
45
+ # Environment facts → CONTEXT_FILE (CLAUDE.md)
46
+ if report.environment_facts:
47
+ content = self._format_environment(report.environment_facts)
48
+ recommendations.append(
49
+ Recommendation(
50
+ target=RecommendationTarget.CONTEXT_FILE,
51
+ section="Environment",
52
+ content=content,
53
+ confidence=min(
54
+ 1.0, sum(f.evidence_count for f in report.environment_facts) / 10
55
+ ),
56
+ evidence_count=sum(f.evidence_count for f in report.environment_facts),
57
+ )
58
+ )
59
+
60
+ # Large files → CONTEXT_FILE
61
+ large_files = [n for n in report.structure_notes if n.category == "large_file"]
62
+ if large_files:
63
+ content = self._format_large_files(large_files)
64
+ recommendations.append(
65
+ Recommendation(
66
+ target=RecommendationTarget.CONTEXT_FILE,
67
+ section="Known Large Files",
68
+ content=content,
69
+ confidence=min(1.0, sum(n.evidence_count for n in large_files) / 5),
70
+ evidence_count=sum(n.evidence_count for n in large_files),
71
+ )
72
+ )
73
+
74
+ # Path corrections → CONTEXT_FILE (these are stable project structure facts)
75
+ path_corrections = [n for n in report.structure_notes if n.category == "path_correction"]
76
+ if path_corrections:
77
+ content = self._format_path_corrections(path_corrections)
78
+ recommendations.append(
79
+ Recommendation(
80
+ target=RecommendationTarget.CONTEXT_FILE,
81
+ section="File Path Corrections",
82
+ content=content,
83
+ confidence=0.9,
84
+ evidence_count=sum(n.evidence_count for n in path_corrections),
85
+ )
86
+ )
87
+
88
+ # Search scope corrections → CONTEXT_FILE
89
+ scope_corrections = [n for n in report.structure_notes if n.category == "search_scope"]
90
+ if scope_corrections:
91
+ content = self._format_scope_corrections(scope_corrections)
92
+ recommendations.append(
93
+ Recommendation(
94
+ target=RecommendationTarget.CONTEXT_FILE,
95
+ section="Search Scope",
96
+ content=content,
97
+ confidence=0.8,
98
+ evidence_count=sum(n.evidence_count for n in scope_corrections),
99
+ )
100
+ )
101
+
102
+ # Command patterns → CONTEXT_FILE (stable project-level facts)
103
+ if report.command_patterns:
104
+ content = self._format_command_patterns(report.command_patterns)
105
+ recommendations.append(
106
+ Recommendation(
107
+ target=RecommendationTarget.CONTEXT_FILE,
108
+ section="Command Patterns",
109
+ content=content,
110
+ confidence=0.9,
111
+ evidence_count=sum(p.evidence_count for p in report.command_patterns),
112
+ )
113
+ )
114
+
115
+ # Missing paths (no correction found) → MEMORY_FILE
116
+ missing_paths = [n for n in report.structure_notes if n.category == "missing_path"]
117
+ if missing_paths:
118
+ content = self._format_missing_paths(missing_paths)
119
+ recommendations.append(
120
+ Recommendation(
121
+ target=RecommendationTarget.MEMORY_FILE,
122
+ section="Known Missing Paths",
123
+ content=content,
124
+ confidence=0.6,
125
+ evidence_count=sum(n.evidence_count for n in missing_paths),
126
+ )
127
+ )
128
+
129
+ # Retry patterns (with specific suggestions) → MEMORY_FILE
130
+ if report.retry_patterns:
131
+ content = self._format_retry_patterns(report.retry_patterns)
132
+ recommendations.append(
133
+ Recommendation(
134
+ target=RecommendationTarget.MEMORY_FILE,
135
+ section="Retry Prevention",
136
+ content=content,
137
+ confidence=0.7,
138
+ evidence_count=sum(p.evidence_count for p in report.retry_patterns),
139
+ )
140
+ )
141
+
142
+ # Permission issues → MEMORY_FILE
143
+ if report.permission_issues:
144
+ content = self._format_permissions(report.permission_issues)
145
+ recommendations.append(
146
+ Recommendation(
147
+ target=RecommendationTarget.MEMORY_FILE,
148
+ section="Permission Notes",
149
+ content=content,
150
+ confidence=0.5,
151
+ evidence_count=len(report.permission_issues),
152
+ )
153
+ )
154
+
155
+ return recommendations
156
+
157
+ def _format_environment(self, facts: list[EnvironmentFact]) -> str:
158
+ lines = []
159
+ for fact in facts:
160
+ wrong = ", ".join(f"`{w}`" for w in fact.wrong_commands[:3])
161
+ lines.append(
162
+ f"- **{fact.category.title()}**: use `{fact.correct_command}` "
163
+ f"(not {wrong} — {fact.evidence_count} failures observed)"
164
+ )
165
+ return "\n".join(lines)
166
+
167
+ def _format_large_files(self, notes: list[StructureNote]) -> str:
168
+ lines = ["Always use `offset` and `limit` parameters with Read for these files:"]
169
+ for note in sorted(notes, key=lambda n: -n.evidence_count):
170
+ lines.append(f"- `{note.path}` ({note.note})")
171
+ return "\n".join(lines)
172
+
173
+ def _format_path_corrections(self, notes: list[StructureNote]) -> str:
174
+ lines = ["These file paths are commonly guessed wrong. Use the correct paths:"]
175
+ for note in sorted(notes, key=lambda n: -n.evidence_count):
176
+ lines.append(f"- `{note.path}` → actually at `{note.correct_path}`")
177
+ return "\n".join(lines)
178
+
179
+ def _format_scope_corrections(self, notes: list[StructureNote]) -> str:
180
+ lines = ["When searching, use these scopes (broader paths work, narrow ones fail):"]
181
+ for note in sorted(notes, key=lambda n: -n.evidence_count):
182
+ lines.append(f"- Don't search `{note.path}` → use `{note.correct_path}` instead")
183
+ return "\n".join(lines)
184
+
185
+ def _format_command_patterns(self, patterns: list[CommandPattern]) -> str:
186
+ lines = []
187
+ for p in sorted(patterns, key=lambda p: -p.evidence_count):
188
+ lines.append(f"- **{p.category}**: {p.explanation}")
189
+ lines.append(f" - Wrong: {p.wrong_pattern}")
190
+ lines.append(f" - Correct: {p.correct_pattern}")
191
+ return "\n".join(lines)
192
+
193
+ def _format_missing_paths(self, notes: list[StructureNote]) -> str:
194
+ lines = []
195
+ for note in sorted(notes, key=lambda n: -n.evidence_count):
196
+ lines.append(f"- `{note.path}` — {note.note}")
197
+ return "\n".join(lines)
198
+
199
+ def _format_retry_patterns(self, patterns: list[RetryPattern]) -> str:
200
+ lines = []
201
+ for p in sorted(patterns, key=lambda p: -p.evidence_count):
202
+ lines.append(f"- {p.description}")
203
+ lines.append(f" → {p.suggestion}")
204
+ return "\n".join(lines)
205
+
206
+ def _format_permissions(self, issues: list[str]) -> str:
207
+ lines = []
208
+ for issue in issues:
209
+ lines.append(f"- {issue}")
210
+ return "\n".join(lines)
211
+
212
+
213
+ # =============================================================================
214
+ # Abstract Writer
215
+ # =============================================================================
216
+
217
+
218
+ class ContextWriter(ABC):
219
+ """Base class for writing recommendations to context/memory files."""
220
+
221
+ @abstractmethod
222
+ def write(
223
+ self,
224
+ recommendations: list[Recommendation],
225
+ project: ProjectInfo,
226
+ dry_run: bool = True,
227
+ ) -> WriteResult: ...
228
+
229
+
230
+ # =============================================================================
231
+ # Write Result
232
+ # =============================================================================
233
+
234
+
235
+ class WriteResult:
236
+ """Result of a write operation."""
237
+
238
+ def __init__(self) -> None:
239
+ self.files_written: list[Path] = []
240
+ self.content_by_file: dict[Path, str] = {}
241
+ self.dry_run: bool = True
242
+
243
+ def add(self, path: Path, content: str) -> None:
244
+ self.files_written.append(path)
245
+ self.content_by_file[path] = content
246
+
247
+
248
+ # =============================================================================
249
+ # Claude Code Writer
250
+ # =============================================================================
251
+
252
+
253
+ class ClaudeCodeWriter(ContextWriter):
254
+ """Writes learned patterns to CLAUDE.md and MEMORY.md for Claude Code."""
255
+
256
+ def write(
257
+ self,
258
+ recommendations: list[Recommendation],
259
+ project: ProjectInfo,
260
+ dry_run: bool = True,
261
+ ) -> WriteResult:
262
+ result = WriteResult()
263
+ result.dry_run = dry_run
264
+
265
+ # Group recommendations by target
266
+ context_recs = [r for r in recommendations if r.target == RecommendationTarget.CONTEXT_FILE]
267
+ memory_recs = [r for r in recommendations if r.target == RecommendationTarget.MEMORY_FILE]
268
+
269
+ # Generate CLAUDE.md content
270
+ if context_recs:
271
+ claude_md_path = self._resolve_context_path(project)
272
+ section_content = self._build_section(context_recs)
273
+ full_content = self._merge_into_file(claude_md_path, section_content)
274
+ result.add(claude_md_path, full_content)
275
+
276
+ if not dry_run:
277
+ claude_md_path.parent.mkdir(parents=True, exist_ok=True)
278
+ claude_md_path.write_text(full_content)
279
+
280
+ # Generate MEMORY.md content
281
+ if memory_recs:
282
+ memory_path = self._resolve_memory_path(project)
283
+ section_content = self._build_section(memory_recs)
284
+ full_content = self._merge_into_file(memory_path, section_content)
285
+ result.add(memory_path, full_content)
286
+
287
+ if not dry_run:
288
+ memory_path.parent.mkdir(parents=True, exist_ok=True)
289
+ memory_path.write_text(full_content)
290
+
291
+ return result
292
+
293
+ def _resolve_context_path(self, project: ProjectInfo) -> Path:
294
+ """Resolve path for CLAUDE.md."""
295
+ if project.context_file:
296
+ return project.context_file
297
+ return project.project_path / "CLAUDE.md"
298
+
299
+ def _resolve_memory_path(self, project: ProjectInfo) -> Path:
300
+ """Resolve path for MEMORY.md."""
301
+ if project.memory_file:
302
+ return project.memory_file
303
+ return project.data_path / "memory" / "MEMORY.md"
304
+
305
+ def _build_section(self, recommendations: list[Recommendation]) -> str:
306
+ """Build the marker-delimited section content."""
307
+ now = datetime.now(timezone.utc).strftime("%Y-%m-%d")
308
+ lines = [
309
+ _MARKER_START,
310
+ "## Headroom Learned Patterns",
311
+ f"*Auto-generated by `headroom learn` on {now} — do not edit manually*",
312
+ "",
313
+ ]
314
+
315
+ for rec in recommendations:
316
+ lines.append(f"### {rec.section}")
317
+ lines.append(rec.content)
318
+ lines.append("")
319
+
320
+ lines.append(_MARKER_END)
321
+ return "\n".join(lines)
322
+
323
+ def _merge_into_file(self, file_path: Path, section: str) -> str:
324
+ """Merge the section into an existing file, replacing any prior section."""
325
+ if file_path.exists():
326
+ existing = file_path.read_text()
327
+ # Replace existing headroom section
328
+ if _MARKER_START in existing:
329
+ return _MARKER_PATTERN.sub(section, existing)
330
+ # Append to end
331
+ return existing.rstrip() + "\n\n" + section + "\n"
332
+ else:
333
+ return section + "\n"
headroom/providers/anthropic.py CHANGED
@@ -487,11 +487,11 @@ class AnthropicProvider(Provider):
487
  info = litellm_get_model_info(model)
488
  if info:
489
  if "max_input_tokens" in info and info["max_input_tokens"] is not None:
490
- limit = info["max_input_tokens"]
491
  self._context_limits[model] = limit
492
  return limit
493
  if "max_tokens" in info and info["max_tokens"] is not None:
494
- limit = info["max_tokens"]
495
  self._context_limits[model] = limit
496
  return limit
497
  except Exception as e:
 
487
  info = litellm_get_model_info(model)
488
  if info:
489
  if "max_input_tokens" in info and info["max_input_tokens"] is not None:
490
+ limit = int(info["max_input_tokens"])
491
  self._context_limits[model] = limit
492
  return limit
493
  if "max_tokens" in info and info["max_tokens"] is not None:
494
+ limit = int(info["max_tokens"])
495
  self._context_limits[model] = limit
496
  return limit
497
  except Exception as e:
tests/test_learn/__init__.py ADDED
File without changes
tests/test_learn/test_analyzer.py ADDED
@@ -0,0 +1,348 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Tests for failure analyzers — generic tool call pattern recognition."""
2
+
3
+ from pathlib import Path
4
+
5
+ from headroom.learn.analyzer import FailureAnalyzer
6
+ from headroom.learn.models import (
7
+ ErrorCategory,
8
+ ProjectInfo,
9
+ SessionData,
10
+ ToolCall,
11
+ )
12
+
13
+
14
+ def _project() -> ProjectInfo:
15
+ return ProjectInfo(
16
+ name="test-project",
17
+ project_path=Path("/tmp/test-project"),
18
+ data_path=Path("/tmp/test-data"),
19
+ )
20
+
21
+
22
+ def _tc(
23
+ name: str = "Bash",
24
+ input_data: dict | None = None,
25
+ output: str = "ok",
26
+ is_error: bool = False,
27
+ error_category: ErrorCategory = ErrorCategory.UNKNOWN,
28
+ msg_index: int = 0,
29
+ ) -> ToolCall:
30
+ return ToolCall(
31
+ name=name,
32
+ tool_call_id=f"tc_{msg_index}",
33
+ input_data=input_data or {},
34
+ output=output,
35
+ is_error=is_error,
36
+ error_category=error_category,
37
+ msg_index=msg_index,
38
+ output_bytes=len(output),
39
+ )
40
+
41
+
42
+ class TestAnalyzerBasics:
43
+ def test_empty_sessions(self):
44
+ analyzer = FailureAnalyzer()
45
+ report = analyzer.analyze(_project(), [])
46
+ assert report.total_calls == 0
47
+ assert report.total_failures == 0
48
+ assert report.failure_rate == 0.0
49
+
50
+ def test_no_failures(self):
51
+ analyzer = FailureAnalyzer()
52
+ sessions = [
53
+ SessionData(
54
+ session_id="s1",
55
+ tool_calls=[_tc(msg_index=i) for i in range(10)],
56
+ )
57
+ ]
58
+ report = analyzer.analyze(_project(), sessions)
59
+ assert report.total_calls == 10
60
+ assert report.total_failures == 0
61
+
62
+ def test_basic_failure_counting(self):
63
+ analyzer = FailureAnalyzer()
64
+ sessions = [
65
+ SessionData(
66
+ session_id="s1",
67
+ tool_calls=[
68
+ _tc(msg_index=0),
69
+ _tc(msg_index=1, is_error=True, output="Error: something broke"),
70
+ _tc(msg_index=2),
71
+ ],
72
+ )
73
+ ]
74
+ report = analyzer.analyze(_project(), sessions)
75
+ assert report.total_calls == 3
76
+ assert report.total_failures == 1
77
+
78
+
79
+ class TestEnvironmentAnalyzer:
80
+ def test_detects_wrong_python(self):
81
+ """Module not found with python3 + successes with uv run → learn correct command."""
82
+ analyzer = FailureAnalyzer()
83
+ sessions = [
84
+ SessionData(
85
+ session_id="s1",
86
+ tool_calls=[
87
+ # Failures with python3
88
+ _tc(
89
+ name="Bash",
90
+ input_data={"command": "python3 -c 'import mylib'"},
91
+ output="ModuleNotFoundError: No module named 'mylib'",
92
+ is_error=True,
93
+ error_category=ErrorCategory.MODULE_NOT_FOUND,
94
+ msg_index=0,
95
+ ),
96
+ _tc(
97
+ name="Bash",
98
+ input_data={"command": "python3 -c 'import mylib'"},
99
+ output="ModuleNotFoundError",
100
+ is_error=True,
101
+ error_category=ErrorCategory.MODULE_NOT_FOUND,
102
+ msg_index=1,
103
+ ),
104
+ # Success with uv run
105
+ _tc(
106
+ name="Bash",
107
+ input_data={"command": "uv run python -c 'import mylib'"},
108
+ output="ok",
109
+ msg_index=2,
110
+ ),
111
+ _tc(
112
+ name="Bash",
113
+ input_data={"command": "uv run python -c 'import mylib'"},
114
+ output="ok",
115
+ msg_index=3,
116
+ ),
117
+ _tc(
118
+ name="Bash",
119
+ input_data={"command": "uv run python -c 'import mylib'"},
120
+ output="ok",
121
+ msg_index=4,
122
+ ),
123
+ ],
124
+ )
125
+ ]
126
+ report = analyzer.analyze(_project(), sessions)
127
+ assert len(report.environment_facts) >= 1
128
+ fact = report.environment_facts[0]
129
+ assert fact.category == "python"
130
+ assert "uv run" in fact.correct_command
131
+ assert "python3" in fact.wrong_commands
132
+
133
+
134
+ class TestStructureAnalyzer:
135
+ def test_detects_missing_paths(self):
136
+ """Files that repeatedly fail Read → learned as missing."""
137
+ analyzer = FailureAnalyzer()
138
+ sessions = [
139
+ SessionData(
140
+ session_id="s1",
141
+ tool_calls=[
142
+ _tc(
143
+ name="Read",
144
+ input_data={"file_path": "/src/missing.py"},
145
+ output="No such file",
146
+ is_error=True,
147
+ error_category=ErrorCategory.FILE_NOT_FOUND,
148
+ msg_index=0,
149
+ ),
150
+ ],
151
+ ),
152
+ SessionData(
153
+ session_id="s2",
154
+ tool_calls=[
155
+ _tc(
156
+ name="Read",
157
+ input_data={"file_path": "/src/missing.py"},
158
+ output="No such file",
159
+ is_error=True,
160
+ error_category=ErrorCategory.FILE_NOT_FOUND,
161
+ msg_index=0,
162
+ ),
163
+ ],
164
+ ),
165
+ ]
166
+ report = analyzer.analyze(_project(), sessions)
167
+ missing = [n for n in report.structure_notes if n.category == "missing_path"]
168
+ assert len(missing) >= 1
169
+ assert "/src/missing.py" in missing[0].path
170
+
171
+ def test_detects_large_files(self):
172
+ """Files that repeatedly trigger too-large errors → learned."""
173
+ analyzer = FailureAnalyzer()
174
+ sessions = [
175
+ SessionData(
176
+ session_id="s1",
177
+ tool_calls=[
178
+ _tc(
179
+ name="Read",
180
+ input_data={"file_path": "/src/huge.py"},
181
+ output="file is too large",
182
+ is_error=True,
183
+ error_category=ErrorCategory.FILE_TOO_LARGE,
184
+ msg_index=0,
185
+ ),
186
+ _tc(
187
+ name="Read",
188
+ input_data={"file_path": "/src/huge.py"},
189
+ output="file is too large",
190
+ is_error=True,
191
+ error_category=ErrorCategory.FILE_TOO_LARGE,
192
+ msg_index=1,
193
+ ),
194
+ ],
195
+ ),
196
+ ]
197
+ report = analyzer.analyze(_project(), sessions)
198
+ large = [n for n in report.structure_notes if n.category == "large_file"]
199
+ assert len(large) >= 1
200
+ assert "/src/huge.py" in large[0].path
201
+
202
+ def test_single_occurrence_not_reported(self):
203
+ """A single file_not_found shouldn't be reported (might be transient)."""
204
+ analyzer = FailureAnalyzer()
205
+ sessions = [
206
+ SessionData(
207
+ session_id="s1",
208
+ tool_calls=[
209
+ _tc(
210
+ name="Read",
211
+ input_data={"file_path": "/src/one_time.py"},
212
+ output="No such file",
213
+ is_error=True,
214
+ error_category=ErrorCategory.FILE_NOT_FOUND,
215
+ msg_index=0,
216
+ ),
217
+ ],
218
+ )
219
+ ]
220
+ report = analyzer.analyze(_project(), sessions)
221
+ missing = [n for n in report.structure_notes if n.category == "missing_path"]
222
+ assert len(missing) == 0
223
+
224
+
225
+ class TestRetryAnalyzer:
226
+ def test_detects_stubborn_retries(self):
227
+ """Same tool failing 3+ times in a row → retry pattern."""
228
+ analyzer = FailureAnalyzer()
229
+ sessions = [
230
+ SessionData(
231
+ session_id="s1",
232
+ tool_calls=[
233
+ _tc(
234
+ name="Bash",
235
+ input_data={"command": "mkdir -p /x"},
236
+ output="auto-denied",
237
+ is_error=True,
238
+ error_category=ErrorCategory.PERMISSION_DENIED,
239
+ msg_index=i,
240
+ )
241
+ for i in range(5)
242
+ ],
243
+ )
244
+ ]
245
+ report = analyzer.analyze(_project(), sessions)
246
+ assert len(report.retry_patterns) >= 1
247
+ pattern = report.retry_patterns[0]
248
+ assert pattern.tool_name == "Bash"
249
+ assert pattern.max_retries_seen >= 5
250
+
251
+ def test_two_failures_not_stubborn(self):
252
+ """Only 2 failures shouldn't trigger a retry pattern (threshold is 3)."""
253
+ analyzer = FailureAnalyzer()
254
+ sessions = [
255
+ SessionData(
256
+ session_id="s1",
257
+ tool_calls=[
258
+ _tc(
259
+ name="Glob",
260
+ output="No matches",
261
+ is_error=True,
262
+ error_category=ErrorCategory.NO_MATCHES,
263
+ msg_index=0,
264
+ ),
265
+ _tc(
266
+ name="Glob",
267
+ output="No matches",
268
+ is_error=True,
269
+ error_category=ErrorCategory.NO_MATCHES,
270
+ msg_index=1,
271
+ ),
272
+ _tc(name="Glob", output="found.py", msg_index=2), # Success breaks streak
273
+ ],
274
+ )
275
+ ]
276
+ report = analyzer.analyze(_project(), sessions)
277
+ assert len(report.retry_patterns) == 0
278
+
279
+
280
+ class TestCrossSessionAnalyzer:
281
+ def test_cross_session_pattern(self):
282
+ """Same failure in 3+ sessions → cross-session pattern."""
283
+ analyzer = FailureAnalyzer()
284
+ sessions = [
285
+ SessionData(
286
+ session_id=f"s{i}",
287
+ tool_calls=[
288
+ _tc(
289
+ name="Read",
290
+ input_data={"file_path": "/docs/RESEARCH.md"},
291
+ output="No such file",
292
+ is_error=True,
293
+ error_category=ErrorCategory.FILE_NOT_FOUND,
294
+ msg_index=0,
295
+ ),
296
+ ],
297
+ )
298
+ for i in range(4)
299
+ ]
300
+ report = analyzer.analyze(_project(), sessions)
301
+ assert len(report.cross_session_patterns) >= 1
302
+ assert any("RESEARCH.md" in p for p in report.cross_session_patterns)
303
+
304
+ def test_two_sessions_not_enough(self):
305
+ """Only 2 sessions shouldn't trigger cross-session (threshold is 3)."""
306
+ analyzer = FailureAnalyzer()
307
+ sessions = [
308
+ SessionData(
309
+ session_id=f"s{i}",
310
+ tool_calls=[
311
+ _tc(
312
+ name="Read",
313
+ input_data={"file_path": "/rare.py"},
314
+ output="No such file",
315
+ is_error=True,
316
+ error_category=ErrorCategory.FILE_NOT_FOUND,
317
+ msg_index=0,
318
+ ),
319
+ ],
320
+ )
321
+ for i in range(2)
322
+ ]
323
+ report = analyzer.analyze(_project(), sessions)
324
+ assert len(report.cross_session_patterns) == 0
325
+
326
+
327
+ class TestPermissionAnalyzer:
328
+ def test_detects_repeated_denials(self):
329
+ """Commands denied 3+ times → permission note."""
330
+ analyzer = FailureAnalyzer()
331
+ sessions = [
332
+ SessionData(
333
+ session_id="s1",
334
+ tool_calls=[
335
+ _tc(
336
+ name="Bash",
337
+ input_data={"command": "mkdir -p /x"},
338
+ output="auto-denied",
339
+ is_error=True,
340
+ error_category=ErrorCategory.PERMISSION_DENIED,
341
+ msg_index=i,
342
+ )
343
+ for i in range(4)
344
+ ],
345
+ )
346
+ ]
347
+ report = analyzer.analyze(_project(), sessions)
348
+ assert len(report.permission_issues) >= 1
tests/test_learn/test_writer.py ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Tests for recommendation writer — marker-based file updates."""
2
+
3
+ from pathlib import Path
4
+
5
+ from headroom.learn.models import ProjectInfo, Recommendation, RecommendationTarget
6
+ from headroom.learn.writer import _MARKER_END, _MARKER_START, ClaudeCodeWriter
7
+
8
+
9
+ def _project(tmp_path: Path) -> ProjectInfo:
10
+ proj_dir = tmp_path / "myproject"
11
+ proj_dir.mkdir()
12
+ data_dir = tmp_path / "data"
13
+ data_dir.mkdir()
14
+ memory_dir = data_dir / "memory"
15
+ memory_dir.mkdir()
16
+ return ProjectInfo(
17
+ name="myproject",
18
+ project_path=proj_dir,
19
+ data_path=data_dir,
20
+ )
21
+
22
+
23
+ def _rec(target: RecommendationTarget, section: str, content: str) -> Recommendation:
24
+ return Recommendation(
25
+ target=target, section=section, content=content, confidence=0.8, evidence_count=5
26
+ )
27
+
28
+
29
+ class TestClaudeCodeWriter:
30
+ def test_dry_run_does_not_write(self, tmp_path):
31
+ proj = _project(tmp_path)
32
+ writer = ClaudeCodeWriter()
33
+ recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- Use uv")]
34
+
35
+ result = writer.write(recs, proj, dry_run=True)
36
+
37
+ assert result.dry_run is True
38
+ assert len(result.files_written) == 1
39
+ # File should NOT exist (dry run)
40
+ claude_md = proj.project_path / "CLAUDE.md"
41
+ assert not claude_md.exists()
42
+
43
+ def test_apply_writes_claude_md(self, tmp_path):
44
+ proj = _project(tmp_path)
45
+ writer = ClaudeCodeWriter()
46
+ recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- Use `uv run python`")]
47
+
48
+ result = writer.write(recs, proj, dry_run=False)
49
+
50
+ assert result.dry_run is False
51
+ claude_md = proj.project_path / "CLAUDE.md"
52
+ assert claude_md.exists()
53
+ content = claude_md.read_text()
54
+ assert "uv run python" in content
55
+ assert _MARKER_START in content
56
+ assert _MARKER_END in content
57
+
58
+ def test_apply_writes_memory_md(self, tmp_path):
59
+ proj = _project(tmp_path)
60
+ writer = ClaudeCodeWriter()
61
+ recs = [_rec(RecommendationTarget.MEMORY_FILE, "Retry Prevention", "- Don't retry globs")]
62
+
63
+ writer.write(recs, proj, dry_run=False)
64
+
65
+ memory_md = proj.data_path / "memory" / "MEMORY.md"
66
+ assert memory_md.exists()
67
+ assert "Don't retry globs" in memory_md.read_text()
68
+
69
+ def test_preserves_existing_claude_md_content(self, tmp_path):
70
+ proj = _project(tmp_path)
71
+ claude_md = proj.project_path / "CLAUDE.md"
72
+ claude_md.write_text("# My Project\n\nExisting instructions here.\n")
73
+
74
+ writer = ClaudeCodeWriter()
75
+ recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- Use uv")]
76
+ writer.write(recs, proj, dry_run=False)
77
+
78
+ content = claude_md.read_text()
79
+ assert "My Project" in content
80
+ assert "Existing instructions here" in content
81
+ assert "Use uv" in content
82
+
83
+ def test_replaces_existing_headroom_section(self, tmp_path):
84
+ proj = _project(tmp_path)
85
+ claude_md = proj.project_path / "CLAUDE.md"
86
+ old_section = (
87
+ f"# My Project\n\n{_MARKER_START}\n## Old Patterns\nold stuff\n{_MARKER_END}\n"
88
+ )
89
+ claude_md.write_text(old_section)
90
+
91
+ writer = ClaudeCodeWriter()
92
+ recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- New stuff")]
93
+ writer.write(recs, proj, dry_run=False)
94
+
95
+ content = claude_md.read_text()
96
+ assert "old stuff" not in content
97
+ assert "New stuff" in content
98
+ assert "My Project" in content
99
+ # Should have exactly one marker pair
100
+ assert content.count(_MARKER_START) == 1
101
+ assert content.count(_MARKER_END) == 1
102
+
103
+ def test_appends_to_existing_memory_md(self, tmp_path):
104
+ proj = _project(tmp_path)
105
+ memory_md = proj.data_path / "memory" / "MEMORY.md"
106
+ memory_md.write_text("# Existing Memory\n\nSome facts.\n")
107
+
108
+ writer = ClaudeCodeWriter()
109
+ recs = [_rec(RecommendationTarget.MEMORY_FILE, "Retry Prevention", "- New pattern")]
110
+ writer.write(recs, proj, dry_run=False)
111
+
112
+ content = memory_md.read_text()
113
+ assert "Existing Memory" in content
114
+ assert "Some facts" in content
115
+ assert "New pattern" in content