Spaces:
Build error
Build error
Commit ·
c14b9ac
1
Parent(s): 2bd9316
Add headroom learn: offline failure learning for coding agents
Browse filesAnalyzes past conversation history to find tool call failure patterns,
correlates each failure with what eventually succeeded, and writes
specific project-level learnings to CLAUDE.md and MEMORY.md.
Key design:
- Success correlation: extracts the diff between failed and successful
inputs as the learning (not generic advice)
- Generic architecture: tool-agnostic ToolCall model with pluggable
Scanner/Writer adapters (Claude Code first, extensible to Cursor/Codex)
- 5 analyzers: Environment, Structure, Commands, Retries, Cross-Session
- Dry-run by default, --apply to write, --all for all projects
Also fixes mypy errors in litellm_callback, asgi, langchain chat_model,
and anthropic provider (AsyncClient typing, ToolCall arg-type, int cast).
- CHANGELOG.md +15 -0
- README.md +13 -0
- docs/learn.md +162 -0
- headroom/cli/learn.py +148 -0
- headroom/cli/main.py +1 -0
- headroom/integrations/asgi.py +1 -1
- headroom/integrations/langchain/chat_model.py +1 -1
- headroom/integrations/litellm_callback.py +1 -1
- headroom/learn/__init__.py +17 -0
- headroom/learn/analyzer.py +618 -0
- headroom/learn/models.py +240 -0
- headroom/learn/scanner.py +358 -0
- headroom/learn/writer.py +333 -0
- headroom/providers/anthropic.py +2 -2
- tests/test_learn/__init__.py +0 -0
- tests/test_learn/test_analyzer.py +348 -0
- tests/test_learn/test_writer.py +115 -0
CHANGELOG.md
CHANGED
|
@@ -8,6 +8,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
| 8 |
## [Unreleased]
|
| 9 |
|
| 10 |
### Added
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
- **any-llm backend** - Route requests through 38+ LLM providers (OpenAI, Mistral, Groq, Ollama, etc.) via [any-llm](https://mozilla-ai.github.io/any-llm/providers/)
|
| 12 |
- Enable with `--backend anyllm --anyllm-provider <provider>`
|
| 13 |
- Install with: `pip install 'headroom-ai[anyllm]'`
|
|
|
|
| 8 |
## [Unreleased]
|
| 9 |
|
| 10 |
### Added
|
| 11 |
+
- **`headroom learn`** — Offline failure learning for coding agents
|
| 12 |
+
- Analyzes past conversation history (Claude Code, extensible to Cursor/Codex)
|
| 13 |
+
- **Success correlation**: for each failure, finds what succeeded after and extracts the specific correction
|
| 14 |
+
- 5 analyzers: Environment, Structure, Command Patterns, Retry Prevention, Cross-Session
|
| 15 |
+
- Writes specific learnings to CLAUDE.md (stable project facts) and MEMORY.md (session patterns)
|
| 16 |
+
- Generic architecture: tool-agnostic `ToolCall` model, pluggable Scanner/Writer adapters
|
| 17 |
+
- Dry-run by default, `--apply` to write, `--all` for all projects
|
| 18 |
+
- Example output: "FirstClassEntity.java is not at axion-formats/ — actually at axion-scala-common/"
|
| 19 |
+
- **Read Lifecycle Management** — Event-driven compression of stale/superseded Read outputs
|
| 20 |
+
- Detects when a Read output becomes stale (file was edited after) or superseded (file was re-read)
|
| 21 |
+
- Replaces stale/superseded content with compact CCR markers, stores originals for retrieval
|
| 22 |
+
- 75% of Read output bytes are provably stale or redundant (from real-world analysis of 66K tool calls)
|
| 23 |
+
- Fresh Reads (latest read, no subsequent edit) are never touched — Edit safety preserved
|
| 24 |
+
- Opt-in via `ReadLifecycleConfig(enabled=True)`, disabled by default
|
| 25 |
+
- Handles both OpenAI and Anthropic message formats
|
| 26 |
- **any-llm backend** - Route requests through 38+ LLM providers (OpenAI, Mistral, Groq, Ollama, etc.) via [any-llm](https://mozilla-ai.github.io/any-llm/providers/)
|
| 27 |
- Enable with `--backend anyllm --anyllm-provider <provider>`
|
| 28 |
- Install with: `pip install 'headroom-ai[anyllm]'`
|
README.md
CHANGED
|
@@ -61,6 +61,16 @@ OPENAI_BASE_URL=http://localhost:8787/v1 cursor
|
|
| 61 |
|
| 62 |
Works with any language, any tool, any framework. One env var. **[Proxy docs](docs/proxy.md)**
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
### Python: One function
|
| 65 |
|
| 66 |
```python
|
|
@@ -231,6 +241,8 @@ flowchart TB
|
|
| 231 |
| **Image Compression** | 40-90% token reduction via trained ML router |
|
| 232 |
| **Memory** | Persistent memory across conversations |
|
| 233 |
| **Compression Hooks** | Customize compression with pre/post hooks |
|
|
|
|
|
|
|
| 234 |
|
| 235 |
---
|
| 236 |
|
|
@@ -276,6 +288,7 @@ Python 3.10+
|
|
| 276 |
| [Memory](docs/memory.md) | Persistent memory |
|
| 277 |
| [Agno](docs/agno.md) | Agno agent framework |
|
| 278 |
| [MCP](docs/mcp.md) | Claude Code subscriptions |
|
|
|
|
| 279 |
| [Configuration](docs/configuration.md) | All options |
|
| 280 |
|
| 281 |
---
|
|
|
|
| 61 |
|
| 62 |
Works with any language, any tool, any framework. One env var. **[Proxy docs](docs/proxy.md)**
|
| 63 |
|
| 64 |
+
### Failure Learning (new)
|
| 65 |
+
|
| 66 |
+
```bash
|
| 67 |
+
headroom learn # Analyze past Claude Code sessions, show recommendations
|
| 68 |
+
headroom learn --apply # Write learnings to CLAUDE.md and MEMORY.md
|
| 69 |
+
headroom learn --all --apply # Learn across all your projects
|
| 70 |
+
```
|
| 71 |
+
|
| 72 |
+
Reads your conversation history, finds every failed tool call, correlates it with what eventually succeeded, and writes specific corrections into your project files. Next session starts smarter. **[Learn docs](docs/learn.md)**
|
| 73 |
+
|
| 74 |
### Python: One function
|
| 75 |
|
| 76 |
```python
|
|
|
|
| 241 |
| **Image Compression** | 40-90% token reduction via trained ML router |
|
| 242 |
| **Memory** | Persistent memory across conversations |
|
| 243 |
| **Compression Hooks** | Customize compression with pre/post hooks |
|
| 244 |
+
| **Read Lifecycle** | Detects stale/superseded Read outputs, replaces with CCR markers |
|
| 245 |
+
| **`headroom learn`** | Analyzes past failures, writes project-specific learnings to CLAUDE.md/MEMORY.md |
|
| 246 |
|
| 247 |
---
|
| 248 |
|
|
|
|
| 288 |
| [Memory](docs/memory.md) | Persistent memory |
|
| 289 |
| [Agno](docs/agno.md) | Agno agent framework |
|
| 290 |
| [MCP](docs/mcp.md) | Claude Code subscriptions |
|
| 291 |
+
| [Learn](docs/learn.md) | Offline failure learning for coding agents |
|
| 292 |
| [Configuration](docs/configuration.md) | All options |
|
| 293 |
|
| 294 |
---
|
docs/learn.md
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Headroom Learn
|
| 2 |
+
|
| 3 |
+
Offline failure learning for coding agents. Analyzes past conversations, finds what went wrong, correlates it with what eventually worked, and writes specific project-level learnings that prevent the same mistakes next session.
|
| 4 |
+
|
| 5 |
+
## Quick Start
|
| 6 |
+
|
| 7 |
+
```bash
|
| 8 |
+
# See recommendations for current project (dry-run, no changes)
|
| 9 |
+
headroom learn
|
| 10 |
+
|
| 11 |
+
# Write recommendations to CLAUDE.md and MEMORY.md
|
| 12 |
+
headroom learn --apply
|
| 13 |
+
|
| 14 |
+
# Analyze a specific project
|
| 15 |
+
headroom learn --project ~/my-project --apply
|
| 16 |
+
|
| 17 |
+
# Analyze all projects
|
| 18 |
+
headroom learn --all --apply
|
| 19 |
+
```
|
| 20 |
+
|
| 21 |
+
## How It Works
|
| 22 |
+
|
| 23 |
+
```
|
| 24 |
+
Past Sessions → Scanner → Analyzer → Writer → CLAUDE.md / MEMORY.md
|
| 25 |
+
│ │ │
|
| 26 |
+
│ │ └─ Writes marker-delimited sections
|
| 27 |
+
│ │ (replaced on re-run, not duplicated)
|
| 28 |
+
│ │
|
| 29 |
+
│ └─ Success Correlation: for each failure,
|
| 30 |
+
│ finds what succeeded and extracts the diff
|
| 31 |
+
│
|
| 32 |
+
└─ Reads ~/.claude/projects/*.jsonl
|
| 33 |
+
(extensible to Cursor, Codex, etc.)
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
### Success Correlation
|
| 37 |
+
|
| 38 |
+
The core innovation. Instead of cataloging failures ("Read failed 5 times"), Headroom finds what the model did to fix each failure:
|
| 39 |
+
|
| 40 |
+
- **Failed**: `Read axion-formats/src/main/java/.../FirstClassEntity.java`
|
| 41 |
+
- **Then succeeded**: `Read axion-scala-common/src/main/scala/.../FirstClassEntity.scala`
|
| 42 |
+
- **Learning**: "`FirstClassEntity` is at `axion-scala-common/`, not `axion-formats/`"
|
| 43 |
+
|
| 44 |
+
This produces specific, actionable corrections — not generic advice.
|
| 45 |
+
|
| 46 |
+
## What It Learns
|
| 47 |
+
|
| 48 |
+
### 1. Environment Facts → CLAUDE.md
|
| 49 |
+
Which runtime commands work vs fail.
|
| 50 |
+
|
| 51 |
+
```markdown
|
| 52 |
+
### Environment
|
| 53 |
+
- **Python**: use `uv run python` (not `python3` — modules not available outside venv)
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
### 2. File Path Corrections → CLAUDE.md
|
| 57 |
+
Wrong paths the model keeps guessing, with the correct locations.
|
| 58 |
+
|
| 59 |
+
```markdown
|
| 60 |
+
### File Path Corrections
|
| 61 |
+
- `axion-common/src/.../AxionSparkConstants.scala`
|
| 62 |
+
→ actually at `axion-spark-common/src/.../AxionSparkConstants.scala`
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
### 3. Search Scope → CLAUDE.md
|
| 66 |
+
Which directories to search in (narrow paths fail, broader ones work).
|
| 67 |
+
|
| 68 |
+
```markdown
|
| 69 |
+
### Search Scope
|
| 70 |
+
- Don't search `axion-model/` → use `axion/` (the repo root)
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
### 4. Command Patterns → CLAUDE.md
|
| 74 |
+
How commands should (and shouldn't) be run.
|
| 75 |
+
|
| 76 |
+
```markdown
|
| 77 |
+
### Command Patterns
|
| 78 |
+
- **user_prefers_manual**: User rejected gradle 18 times — show the command, don't execute
|
| 79 |
+
- **python_runtime**: Use `uv run python` not `python3` (ModuleNotFoundError)
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
### 5. Known Large Files → CLAUDE.md
|
| 83 |
+
Files that need `offset`/`limit` with Read.
|
| 84 |
+
|
| 85 |
+
```markdown
|
| 86 |
+
### Known Large Files
|
| 87 |
+
- `proxy/server.py` (~8000 lines) — always use offset/limit
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
### 6. Retry Prevention → MEMORY.md
|
| 91 |
+
Specific suggestions derived from actual corrections.
|
| 92 |
+
|
| 93 |
+
### 7. Permission Notes → MEMORY.md
|
| 94 |
+
Commands repeatedly rejected — model should suggest them to the user instead.
|
| 95 |
+
|
| 96 |
+
## Where Learnings Go
|
| 97 |
+
|
| 98 |
+
| Pattern | Destination | Why |
|
| 99 |
+
|---------|-------------|-----|
|
| 100 |
+
| Environment, paths, search scope, commands, large files | **CLAUDE.md** | Stable project facts, version-controllable |
|
| 101 |
+
| Missing paths, retry patterns, permissions | **MEMORY.md** | May change, agent-specific |
|
| 102 |
+
|
| 103 |
+
CLAUDE.md lives in your project directory. MEMORY.md lives in `~/.claude/projects/*/memory/`.
|
| 104 |
+
|
| 105 |
+
## Marker-Based Updates
|
| 106 |
+
|
| 107 |
+
Headroom manages a clearly-delimited section in each file:
|
| 108 |
+
|
| 109 |
+
```markdown
|
| 110 |
+
<!-- headroom:learn:start -->
|
| 111 |
+
## Headroom Learned Patterns
|
| 112 |
+
*Auto-generated by `headroom learn` — do not edit manually*
|
| 113 |
+
...
|
| 114 |
+
<!-- headroom:learn:end -->
|
| 115 |
+
```
|
| 116 |
+
|
| 117 |
+
On re-run, only the content between markers is replaced. Your existing file content is preserved.
|
| 118 |
+
|
| 119 |
+
## Architecture
|
| 120 |
+
|
| 121 |
+
```
|
| 122 |
+
Scanner (adapter) → Analyzer (generic) → Writer (adapter)
|
| 123 |
+
├── ClaudeCodeScanner ├── EnvironmentAnalyzer ├── ClaudeCodeWriter
|
| 124 |
+
├── (CursorScanner) ├── StructureAnalyzer ├── (CursorWriter)
|
| 125 |
+
└── (GenericScanner) ├── CommandAnalyzer └── (GenericWriter)
|
| 126 |
+
├── RetryAnalyzer
|
| 127 |
+
└── CrossSessionAnalyzer
|
| 128 |
+
```
|
| 129 |
+
|
| 130 |
+
**Scanners** read tool-specific log formats and produce normalized `ToolCall` sequences.
|
| 131 |
+
**Analyzers** work on `ToolCall` — same analysis for any agent system.
|
| 132 |
+
**Writers** output to tool-specific context injection mechanisms.
|
| 133 |
+
|
| 134 |
+
To add support for a new agent (e.g., Cursor):
|
| 135 |
+
1. Write `CursorScanner(ConversationScanner)` — reads Cursor's log format
|
| 136 |
+
2. Write `CursorWriter(ContextWriter)` — writes to `.cursorrules`
|
| 137 |
+
3. Same analyzers, same models, same recommendations
|
| 138 |
+
|
| 139 |
+
## CLI Reference
|
| 140 |
+
|
| 141 |
+
```
|
| 142 |
+
headroom learn [OPTIONS]
|
| 143 |
+
|
| 144 |
+
Options:
|
| 145 |
+
--project PATH Project directory to analyze (default: current directory)
|
| 146 |
+
--all Analyze all discovered projects
|
| 147 |
+
--apply Write recommendations (default: dry-run)
|
| 148 |
+
--claude-dir PATH Path to .claude directory (default: ~/.claude)
|
| 149 |
+
```
|
| 150 |
+
|
| 151 |
+
## Real-World Results
|
| 152 |
+
|
| 153 |
+
Tested on 67,583 tool calls across 23 projects:
|
| 154 |
+
|
| 155 |
+
| Metric | Value |
|
| 156 |
+
|--------|-------|
|
| 157 |
+
| Failure rate | 7.5% (5,066 failures) |
|
| 158 |
+
| Corrections extracted | 164 per project (avg) |
|
| 159 |
+
| Specific path corrections | 22 (axion project) |
|
| 160 |
+
| Search scope corrections | 24 (axion project) |
|
| 161 |
+
| Command patterns learned | 5 (axion project) |
|
| 162 |
+
| Estimated preventable waste | ~27 MB across corpus |
|
headroom/cli/learn.py
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""CLI commands for Headroom Learn — offline failure learning."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
|
| 7 |
+
import click
|
| 8 |
+
|
| 9 |
+
from .main import main
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
@main.command()
|
| 13 |
+
@click.option(
|
| 14 |
+
"--project",
|
| 15 |
+
type=click.Path(exists=True, path_type=Path),
|
| 16 |
+
default=None,
|
| 17 |
+
help="Project directory to analyze. Defaults to current directory.",
|
| 18 |
+
)
|
| 19 |
+
@click.option(
|
| 20 |
+
"--all",
|
| 21 |
+
"analyze_all",
|
| 22 |
+
is_flag=True,
|
| 23 |
+
default=False,
|
| 24 |
+
help="Analyze all discovered projects.",
|
| 25 |
+
)
|
| 26 |
+
@click.option(
|
| 27 |
+
"--apply",
|
| 28 |
+
is_flag=True,
|
| 29 |
+
default=False,
|
| 30 |
+
help="Write recommendations to CLAUDE.md / MEMORY.md (default: dry-run).",
|
| 31 |
+
)
|
| 32 |
+
@click.option(
|
| 33 |
+
"--claude-dir",
|
| 34 |
+
type=click.Path(path_type=Path),
|
| 35 |
+
default=None,
|
| 36 |
+
help="Path to .claude directory. Defaults to ~/.claude.",
|
| 37 |
+
)
|
| 38 |
+
def learn(
|
| 39 |
+
project: Path | None,
|
| 40 |
+
analyze_all: bool,
|
| 41 |
+
apply: bool,
|
| 42 |
+
claude_dir: Path | None,
|
| 43 |
+
) -> None:
|
| 44 |
+
"""Learn from past tool call failures to prevent future ones.
|
| 45 |
+
|
| 46 |
+
Analyzes conversation history to find failure patterns (wrong paths,
|
| 47 |
+
missing modules, stubborn retries) and generates context that prevents
|
| 48 |
+
them from recurring.
|
| 49 |
+
|
| 50 |
+
\b
|
| 51 |
+
Examples:
|
| 52 |
+
headroom learn # Analyze current project (dry-run)
|
| 53 |
+
headroom learn --apply # Write recommendations
|
| 54 |
+
headroom learn --all # Analyze all projects
|
| 55 |
+
headroom learn --project ~/myapp # Analyze specific project
|
| 56 |
+
"""
|
| 57 |
+
from ..learn.analyzer import FailureAnalyzer
|
| 58 |
+
from ..learn.scanner import ClaudeCodeScanner
|
| 59 |
+
from ..learn.writer import ClaudeCodeWriter, Recommender
|
| 60 |
+
|
| 61 |
+
scanner = ClaudeCodeScanner(claude_dir=claude_dir)
|
| 62 |
+
analyzer = FailureAnalyzer()
|
| 63 |
+
recommender = Recommender()
|
| 64 |
+
writer = ClaudeCodeWriter()
|
| 65 |
+
|
| 66 |
+
# Discover projects
|
| 67 |
+
all_projects = scanner.discover_projects()
|
| 68 |
+
|
| 69 |
+
if not all_projects:
|
| 70 |
+
click.echo("No projects found in ~/.claude/projects/")
|
| 71 |
+
return
|
| 72 |
+
|
| 73 |
+
# Filter to target project(s)
|
| 74 |
+
if analyze_all:
|
| 75 |
+
targets = all_projects
|
| 76 |
+
elif project:
|
| 77 |
+
resolved = project.resolve()
|
| 78 |
+
targets = [p for p in all_projects if p.project_path == resolved]
|
| 79 |
+
if not targets:
|
| 80 |
+
click.echo(f"Project not found: {resolved}")
|
| 81 |
+
click.echo(f"Available projects: {', '.join(p.name for p in all_projects)}")
|
| 82 |
+
return
|
| 83 |
+
else:
|
| 84 |
+
# Auto-detect from cwd
|
| 85 |
+
cwd = Path.cwd().resolve()
|
| 86 |
+
targets = [p for p in all_projects if p.project_path == cwd]
|
| 87 |
+
if not targets:
|
| 88 |
+
# Try parent directories
|
| 89 |
+
for parent in cwd.parents:
|
| 90 |
+
targets = [p for p in all_projects if p.project_path == parent]
|
| 91 |
+
if targets:
|
| 92 |
+
break
|
| 93 |
+
if not targets:
|
| 94 |
+
click.echo(f"No project data found for {cwd}")
|
| 95 |
+
click.echo("Try: headroom learn --project <path> or headroom learn --all")
|
| 96 |
+
click.echo("\nAvailable projects:")
|
| 97 |
+
for p in all_projects[:10]:
|
| 98 |
+
click.echo(f" {p.name:30s} {p.project_path}")
|
| 99 |
+
return
|
| 100 |
+
|
| 101 |
+
# Analyze each target
|
| 102 |
+
for proj in targets:
|
| 103 |
+
click.echo(f"\n{'=' * 60}")
|
| 104 |
+
click.echo(f"Project: {proj.name}")
|
| 105 |
+
click.echo(f"Path: {proj.project_path}")
|
| 106 |
+
click.echo(f"{'=' * 60}")
|
| 107 |
+
|
| 108 |
+
sessions = scanner.scan_project(proj)
|
| 109 |
+
if not sessions:
|
| 110 |
+
click.echo(" No conversation data found.")
|
| 111 |
+
continue
|
| 112 |
+
|
| 113 |
+
report = analyzer.analyze(proj, sessions)
|
| 114 |
+
|
| 115 |
+
# Print summary
|
| 116 |
+
click.echo(f"\n Sessions analyzed: {report.total_sessions}")
|
| 117 |
+
click.echo(f" Total tool calls: {report.total_calls}")
|
| 118 |
+
click.echo(f" Failed calls: {report.total_failures} ({report.failure_rate:.1%})")
|
| 119 |
+
click.echo(f" Waste bytes: {report.waste_bytes / 1024:.0f} KB")
|
| 120 |
+
|
| 121 |
+
if report.failure_rate == 0:
|
| 122 |
+
click.echo("\n No failures found. Nothing to learn.")
|
| 123 |
+
continue
|
| 124 |
+
|
| 125 |
+
# Generate recommendations
|
| 126 |
+
recommendations = recommender.recommend(report)
|
| 127 |
+
|
| 128 |
+
if not recommendations:
|
| 129 |
+
click.echo("\n No actionable patterns found.")
|
| 130 |
+
continue
|
| 131 |
+
|
| 132 |
+
click.echo(f"\n Recommendations: {len(recommendations)}")
|
| 133 |
+
|
| 134 |
+
# Write (or dry-run)
|
| 135 |
+
result = writer.write(recommendations, proj, dry_run=not apply)
|
| 136 |
+
|
| 137 |
+
for file_path, content in result.content_by_file.items():
|
| 138 |
+
click.echo(f"\n {'[WOULD WRITE]' if result.dry_run else '[WROTE]'} {file_path}")
|
| 139 |
+
click.echo(f" {'─' * 50}")
|
| 140 |
+
# Show content preview (indented)
|
| 141 |
+
for line in content.split("\n"):
|
| 142 |
+
if line.startswith("<!-- headroom"):
|
| 143 |
+
continue # Skip markers in display
|
| 144 |
+
click.echo(f" {line}")
|
| 145 |
+
click.echo(f" {'─' * 50}")
|
| 146 |
+
|
| 147 |
+
if result.dry_run:
|
| 148 |
+
click.echo("\n Dry run — no files modified. Use --apply to write.")
|
headroom/cli/main.py
CHANGED
|
@@ -35,6 +35,7 @@ def _register_commands() -> None:
|
|
| 35 |
"""Register all subcommand groups."""
|
| 36 |
from . import (
|
| 37 |
evals, # noqa: F401
|
|
|
|
| 38 |
mcp, # noqa: F401
|
| 39 |
memory, # noqa: F401
|
| 40 |
proxy, # noqa: F401
|
|
|
|
| 35 |
"""Register all subcommand groups."""
|
| 36 |
from . import (
|
| 37 |
evals, # noqa: F401
|
| 38 |
+
learn, # noqa: F401
|
| 39 |
mcp, # noqa: F401
|
| 40 |
memory, # noqa: F401
|
| 41 |
proxy, # noqa: F401
|
headroom/integrations/asgi.py
CHANGED
|
@@ -80,7 +80,7 @@ class CompressionMiddleware:
|
|
| 80 |
self._api_url = (
|
| 81 |
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
|
| 82 |
).rstrip("/")
|
| 83 |
-
self._client = None # Lazy-initialized httpx.AsyncClient
|
| 84 |
|
| 85 |
@property
|
| 86 |
def cloud_mode(self) -> bool:
|
|
|
|
| 80 |
self._api_url = (
|
| 81 |
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
|
| 82 |
).rstrip("/")
|
| 83 |
+
self._client: Any = None # Lazy-initialized httpx.AsyncClient
|
| 84 |
|
| 85 |
@property
|
| 86 |
def cloud_mode(self) -> bool:
|
headroom/integrations/langchain/chat_model.py
CHANGED
|
@@ -79,7 +79,7 @@ def _check_langchain_available() -> None:
|
|
| 79 |
)
|
| 80 |
|
| 81 |
|
| 82 |
-
def _tool_call_args_to_json(tc: dict[str, Any]) -> str:
|
| 83 |
"""Normalize tool call arguments to JSON string for OpenAI format.
|
| 84 |
|
| 85 |
LangChain can provide 'args' (dict) or 'arguments' (str) depending on source.
|
|
|
|
| 79 |
)
|
| 80 |
|
| 81 |
|
| 82 |
+
def _tool_call_args_to_json(tc: dict[str, Any] | Any) -> str:
|
| 83 |
"""Normalize tool call arguments to JSON string for OpenAI format.
|
| 84 |
|
| 85 |
LangChain can provide 'args' (dict) or 'arguments' (str) depending on source.
|
headroom/integrations/litellm_callback.py
CHANGED
|
@@ -69,7 +69,7 @@ class HeadroomCallback:
|
|
| 69 |
self._api_url = (
|
| 70 |
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
|
| 71 |
).rstrip("/")
|
| 72 |
-
self._client = None # Lazy-initialized httpx.AsyncClient
|
| 73 |
|
| 74 |
@property
|
| 75 |
def total_tokens_saved(self) -> int:
|
|
|
|
| 69 |
self._api_url = (
|
| 70 |
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
|
| 71 |
).rstrip("/")
|
| 72 |
+
self._client: Any = None # Lazy-initialized httpx.AsyncClient
|
| 73 |
|
| 74 |
@property
|
| 75 |
def total_tokens_saved(self) -> int:
|
headroom/learn/__init__.py
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Headroom Learn — offline failure learning for coding agents.
|
| 2 |
+
|
| 3 |
+
Analyzes conversation logs to find tool call failure patterns and generates
|
| 4 |
+
context (CLAUDE.md, MEMORY.md, .cursorrules, etc.) that prevents future failures.
|
| 5 |
+
|
| 6 |
+
Architecture:
|
| 7 |
+
Scanner (adapter) → Analyzer (generic) → Writer (adapter)
|
| 8 |
+
├── ClaudeCodeScanner ├── EnvironmentAnalyzer ├── ClaudeCodeWriter
|
| 9 |
+
├── CursorScanner ├── StructureAnalyzer ├── CursorWriter
|
| 10 |
+
└── GenericScanner ├── RetryAnalyzer └── GenericWriter
|
| 11 |
+
├── PermissionAnalyzer
|
| 12 |
+
└── CrossSessionAnalyzer
|
| 13 |
+
|
| 14 |
+
Scanners read tool-specific log formats and produce normalized ToolCall sequences.
|
| 15 |
+
Analyzers work on ToolCall — same analysis for any agent system.
|
| 16 |
+
Writers output to tool-specific context injection mechanisms.
|
| 17 |
+
"""
|
headroom/learn/analyzer.py
ADDED
|
@@ -0,0 +1,618 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Failure analyzers with success correlation.
|
| 2 |
+
|
| 3 |
+
The core insight: don't just catalog failures — find what SUCCEEDED after
|
| 4 |
+
each failure. The diff between failed input and successful input is the
|
| 5 |
+
actual learning.
|
| 6 |
+
|
| 7 |
+
All analyzers work on normalized ToolCall sequences. They are tool-agnostic:
|
| 8 |
+
same analysis works for Claude Code, Cursor, Codex, or any agent with tool calls.
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import os
|
| 14 |
+
import re
|
| 15 |
+
from collections import Counter, defaultdict
|
| 16 |
+
|
| 17 |
+
from .models import (
|
| 18 |
+
AnalysisReport,
|
| 19 |
+
CommandPattern,
|
| 20 |
+
Correction,
|
| 21 |
+
EnvironmentFact,
|
| 22 |
+
ErrorCategory,
|
| 23 |
+
ProjectInfo,
|
| 24 |
+
RetryPattern,
|
| 25 |
+
SessionData,
|
| 26 |
+
StructureNote,
|
| 27 |
+
ToolCall,
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
# How many messages ahead to look for a success after a failure
|
| 31 |
+
_CORRECTION_WINDOW = 10
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
class FailureAnalyzer:
|
| 35 |
+
"""Runs all analyzers on tool call data and produces an AnalysisReport."""
|
| 36 |
+
|
| 37 |
+
def analyze(self, project: ProjectInfo, sessions: list[SessionData]) -> AnalysisReport:
|
| 38 |
+
all_calls = [tc for s in sessions for tc in s.tool_calls]
|
| 39 |
+
failed_calls = [tc for tc in all_calls if tc.is_error]
|
| 40 |
+
|
| 41 |
+
report = AnalysisReport(
|
| 42 |
+
project=project,
|
| 43 |
+
total_calls=len(all_calls),
|
| 44 |
+
total_failures=len(failed_calls),
|
| 45 |
+
total_sessions=len(sessions),
|
| 46 |
+
waste_bytes=sum(tc.output_bytes for tc in failed_calls),
|
| 47 |
+
)
|
| 48 |
+
|
| 49 |
+
# Phase 1: Extract failure→success corrections (the core learning)
|
| 50 |
+
report.corrections = _extract_corrections(sessions)
|
| 51 |
+
|
| 52 |
+
# Phase 2: Analyze specific dimensions using corrections + raw data
|
| 53 |
+
report.environment_facts = _analyze_environment(sessions)
|
| 54 |
+
report.structure_notes = _analyze_structure(sessions, report.corrections)
|
| 55 |
+
report.command_patterns = _analyze_commands(sessions, report.corrections)
|
| 56 |
+
report.retry_patterns = _analyze_retries(sessions, report.corrections)
|
| 57 |
+
report.permission_issues = _analyze_permissions(sessions)
|
| 58 |
+
report.cross_session_patterns = _analyze_cross_session(sessions)
|
| 59 |
+
|
| 60 |
+
return report
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
# =============================================================================
|
| 64 |
+
# Success Correlation: The Core Learning Primitive
|
| 65 |
+
# =============================================================================
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
def _extract_corrections(sessions: list[SessionData]) -> list[Correction]:
|
| 69 |
+
"""For each failure, find the next success of the same tool type.
|
| 70 |
+
|
| 71 |
+
The pair (failed_input, success_input) is a Correction — the model
|
| 72 |
+
learned something and corrected its approach.
|
| 73 |
+
"""
|
| 74 |
+
corrections: list[Correction] = []
|
| 75 |
+
|
| 76 |
+
for session in sessions:
|
| 77 |
+
calls = session.tool_calls
|
| 78 |
+
for i, tc in enumerate(calls):
|
| 79 |
+
if not tc.is_error:
|
| 80 |
+
continue
|
| 81 |
+
# Skip sibling errors (cascades, not real failures)
|
| 82 |
+
if tc.error_category == ErrorCategory.SIBLING_ERROR:
|
| 83 |
+
continue
|
| 84 |
+
# Look forward for a success of the same tool
|
| 85 |
+
for j in range(i + 1, min(i + _CORRECTION_WINDOW, len(calls))):
|
| 86 |
+
candidate = calls[j]
|
| 87 |
+
if candidate.name != tc.name or candidate.is_error:
|
| 88 |
+
continue
|
| 89 |
+
# Found a success — is the input meaningfully different?
|
| 90 |
+
if candidate.input_data == tc.input_data:
|
| 91 |
+
continue # Exact same input succeeded (transient error)
|
| 92 |
+
corrections.append(
|
| 93 |
+
Correction(
|
| 94 |
+
tool_name=tc.name,
|
| 95 |
+
failed_input=tc.input_data,
|
| 96 |
+
success_input=candidate.input_data,
|
| 97 |
+
error_category=tc.error_category,
|
| 98 |
+
session_id=session.session_id,
|
| 99 |
+
)
|
| 100 |
+
)
|
| 101 |
+
break
|
| 102 |
+
|
| 103 |
+
return corrections
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
# =============================================================================
|
| 107 |
+
# Environment Analyzer (uses corrections for python/build tool detection)
|
| 108 |
+
# =============================================================================
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def _analyze_environment(sessions: list[SessionData]) -> list[EnvironmentFact]:
|
| 112 |
+
"""Detect which runtime commands work vs fail."""
|
| 113 |
+
facts: list[EnvironmentFact] = []
|
| 114 |
+
|
| 115 |
+
python_failures: Counter[str] = Counter()
|
| 116 |
+
python_successes: Counter[str] = Counter()
|
| 117 |
+
python_sessions: dict[str, set[str]] = defaultdict(set)
|
| 118 |
+
|
| 119 |
+
for session in sessions:
|
| 120 |
+
for tc in session.tool_calls:
|
| 121 |
+
if tc.name not in ("Bash", "bash"):
|
| 122 |
+
continue
|
| 123 |
+
cmd = tc.input_data.get("command", "")
|
| 124 |
+
if not cmd:
|
| 125 |
+
continue
|
| 126 |
+
|
| 127 |
+
if tc.is_error and tc.error_category == ErrorCategory.MODULE_NOT_FOUND:
|
| 128 |
+
prefix = _extract_python_command(cmd)
|
| 129 |
+
if prefix:
|
| 130 |
+
python_failures[prefix] += 1
|
| 131 |
+
python_sessions[prefix].add(session.session_id)
|
| 132 |
+
elif not tc.is_error:
|
| 133 |
+
prefix = _extract_python_command(cmd)
|
| 134 |
+
if prefix:
|
| 135 |
+
python_successes[prefix] += 1
|
| 136 |
+
|
| 137 |
+
if python_failures:
|
| 138 |
+
wrong = sorted(python_failures.keys(), key=lambda x: -python_failures[x])
|
| 139 |
+
correct = None
|
| 140 |
+
for cmd, _count in python_successes.most_common():
|
| 141 |
+
if cmd not in python_failures or python_successes[cmd] > python_failures[cmd] * 2:
|
| 142 |
+
correct = cmd
|
| 143 |
+
break
|
| 144 |
+
if correct and wrong:
|
| 145 |
+
total_evidence = sum(python_failures[w] for w in wrong)
|
| 146 |
+
total_sessions = len(set().union(*(python_sessions[w] for w in wrong)))
|
| 147 |
+
facts.append(
|
| 148 |
+
EnvironmentFact(
|
| 149 |
+
category="python",
|
| 150 |
+
correct_command=correct,
|
| 151 |
+
wrong_commands=wrong[:5],
|
| 152 |
+
evidence_count=total_evidence,
|
| 153 |
+
sessions_seen=total_sessions,
|
| 154 |
+
)
|
| 155 |
+
)
|
| 156 |
+
|
| 157 |
+
return facts
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
# =============================================================================
|
| 161 |
+
# Structure Analyzer (uses corrections to learn correct paths)
|
| 162 |
+
# =============================================================================
|
| 163 |
+
|
| 164 |
+
|
| 165 |
+
def _analyze_structure(
|
| 166 |
+
sessions: list[SessionData], corrections: list[Correction]
|
| 167 |
+
) -> list[StructureNote]:
|
| 168 |
+
"""Find file structure issues and learn correct paths from corrections."""
|
| 169 |
+
notes: list[StructureNote] = []
|
| 170 |
+
|
| 171 |
+
# 1. Path corrections: wrong path → correct path (from success correlation)
|
| 172 |
+
path_corrections: dict[str, Counter[str]] = defaultdict(Counter)
|
| 173 |
+
for c in corrections:
|
| 174 |
+
if c.tool_name not in ("Read", "read"):
|
| 175 |
+
continue
|
| 176 |
+
if c.error_category != ErrorCategory.FILE_NOT_FOUND:
|
| 177 |
+
continue
|
| 178 |
+
failed_path = c.failed_input.get("file_path", "")
|
| 179 |
+
success_path = c.success_input.get("file_path", "")
|
| 180 |
+
if failed_path and success_path and failed_path != success_path:
|
| 181 |
+
path_corrections[failed_path][success_path] += 1
|
| 182 |
+
|
| 183 |
+
for wrong_path, correct_paths in path_corrections.items():
|
| 184 |
+
best_correct, count = correct_paths.most_common(1)[0]
|
| 185 |
+
# Make paths relative to project for readability
|
| 186 |
+
wrong_short = _shorten_path(wrong_path)
|
| 187 |
+
correct_short = _shorten_path(best_correct)
|
| 188 |
+
notes.append(
|
| 189 |
+
StructureNote(
|
| 190 |
+
category="path_correction",
|
| 191 |
+
path=wrong_short,
|
| 192 |
+
correct_path=correct_short,
|
| 193 |
+
note=f"Not at `{wrong_short}` → actually at `{correct_short}`",
|
| 194 |
+
evidence_count=count,
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
|
| 198 |
+
# 2. Grep scope corrections: narrow path → broader path worked
|
| 199 |
+
scope_corrections: dict[str, Counter[str]] = defaultdict(Counter)
|
| 200 |
+
for c in corrections:
|
| 201 |
+
if c.tool_name not in ("Grep", "grep"):
|
| 202 |
+
continue
|
| 203 |
+
failed_path = c.failed_input.get("path", "")
|
| 204 |
+
success_path = c.success_input.get("path", "")
|
| 205 |
+
if failed_path and success_path and failed_path != success_path:
|
| 206 |
+
scope_corrections[_shorten_path(failed_path)][_shorten_path(success_path)] += 1
|
| 207 |
+
|
| 208 |
+
for wrong_scope, correct_scopes in scope_corrections.items():
|
| 209 |
+
best_scope, count = correct_scopes.most_common(1)[0]
|
| 210 |
+
notes.append(
|
| 211 |
+
StructureNote(
|
| 212 |
+
category="search_scope",
|
| 213 |
+
path=wrong_scope,
|
| 214 |
+
correct_path=best_scope,
|
| 215 |
+
note=f"Grep fails at `{wrong_scope}` → use `{best_scope}` instead",
|
| 216 |
+
evidence_count=count,
|
| 217 |
+
)
|
| 218 |
+
)
|
| 219 |
+
|
| 220 |
+
# 3. Large files (from raw failures, no correction needed)
|
| 221 |
+
large_files: Counter[str] = Counter()
|
| 222 |
+
large_sessions: dict[str, set[str]] = defaultdict(set)
|
| 223 |
+
for session in sessions:
|
| 224 |
+
for tc in session.tool_calls:
|
| 225 |
+
if (
|
| 226 |
+
tc.name in ("Read", "read")
|
| 227 |
+
and tc.is_error
|
| 228 |
+
and tc.error_category == ErrorCategory.FILE_TOO_LARGE
|
| 229 |
+
):
|
| 230 |
+
path = tc.input_data.get("file_path", "")
|
| 231 |
+
if path:
|
| 232 |
+
short = _shorten_path(path)
|
| 233 |
+
large_files[short] += 1
|
| 234 |
+
large_sessions[short].add(session.session_id)
|
| 235 |
+
|
| 236 |
+
for path, count in large_files.most_common(10):
|
| 237 |
+
if count < 2:
|
| 238 |
+
break
|
| 239 |
+
notes.append(
|
| 240 |
+
StructureNote(
|
| 241 |
+
category="large_file",
|
| 242 |
+
path=path,
|
| 243 |
+
note=f"Too large for full read — always use offset/limit ({count} failures, {len(large_sessions[path])} sessions)",
|
| 244 |
+
evidence_count=count,
|
| 245 |
+
sessions_seen=len(large_sessions[path]),
|
| 246 |
+
)
|
| 247 |
+
)
|
| 248 |
+
|
| 249 |
+
# 4. Persistent missing paths (no correction found — file truly doesn't exist)
|
| 250 |
+
missing_no_correction: Counter[str] = Counter()
|
| 251 |
+
missing_sessions: dict[str, set[str]] = defaultdict(set)
|
| 252 |
+
corrected_paths = set(path_corrections.keys())
|
| 253 |
+
for session in sessions:
|
| 254 |
+
for tc in session.tool_calls:
|
| 255 |
+
if (
|
| 256 |
+
tc.name in ("Read", "read")
|
| 257 |
+
and tc.is_error
|
| 258 |
+
and tc.error_category == ErrorCategory.FILE_NOT_FOUND
|
| 259 |
+
):
|
| 260 |
+
path = tc.input_data.get("file_path", "")
|
| 261 |
+
if path and path not in corrected_paths:
|
| 262 |
+
missing_no_correction[path] += 1
|
| 263 |
+
missing_sessions[path].add(session.session_id)
|
| 264 |
+
|
| 265 |
+
for path, count in missing_no_correction.most_common(10):
|
| 266 |
+
if count < 2:
|
| 267 |
+
break
|
| 268 |
+
short = _shorten_path(path)
|
| 269 |
+
notes.append(
|
| 270 |
+
StructureNote(
|
| 271 |
+
category="missing_path",
|
| 272 |
+
path=short,
|
| 273 |
+
note=f"Does not exist ({count} attempts, {len(missing_sessions[path])} sessions)",
|
| 274 |
+
evidence_count=count,
|
| 275 |
+
sessions_seen=len(missing_sessions[path]),
|
| 276 |
+
)
|
| 277 |
+
)
|
| 278 |
+
|
| 279 |
+
return notes
|
| 280 |
+
|
| 281 |
+
|
| 282 |
+
# =============================================================================
|
| 283 |
+
# Command Pattern Analyzer (uses corrections to learn command patterns)
|
| 284 |
+
# =============================================================================
|
| 285 |
+
|
| 286 |
+
|
| 287 |
+
def _analyze_commands(
|
| 288 |
+
sessions: list[SessionData], corrections: list[Correction]
|
| 289 |
+
) -> list[CommandPattern]:
|
| 290 |
+
"""Learn specific command patterns from Bash failure→success corrections."""
|
| 291 |
+
patterns: list[CommandPattern] = []
|
| 292 |
+
|
| 293 |
+
# Analyze Bash corrections
|
| 294 |
+
bash_corrections = [c for c in corrections if c.tool_name in ("Bash", "bash")]
|
| 295 |
+
|
| 296 |
+
# Group by error category to find patterns
|
| 297 |
+
by_category: dict[ErrorCategory, list[Correction]] = defaultdict(list)
|
| 298 |
+
for c in bash_corrections:
|
| 299 |
+
by_category[c.error_category].append(c)
|
| 300 |
+
|
| 301 |
+
# User-rejected commands: model should suggest, not execute
|
| 302 |
+
rejected = by_category.get(ErrorCategory.USER_REJECTED, [])
|
| 303 |
+
if rejected:
|
| 304 |
+
# Find the most commonly rejected command patterns
|
| 305 |
+
rejected_cmds: Counter[str] = Counter()
|
| 306 |
+
for c in rejected:
|
| 307 |
+
cmd = c.failed_input.get("command", "")
|
| 308 |
+
base = _extract_command_signature(cmd)
|
| 309 |
+
if base:
|
| 310 |
+
rejected_cmds[base] += 1
|
| 311 |
+
|
| 312 |
+
for cmd_sig, count in rejected_cmds.most_common(5):
|
| 313 |
+
if count < 2:
|
| 314 |
+
break
|
| 315 |
+
patterns.append(
|
| 316 |
+
CommandPattern(
|
| 317 |
+
category="user_prefers_manual",
|
| 318 |
+
wrong_pattern=f"Executing: {cmd_sig}",
|
| 319 |
+
correct_pattern="Show the command to the user and let them run it",
|
| 320 |
+
explanation=f"User rejected this command {count} times — they prefer to run it themselves",
|
| 321 |
+
evidence_count=count,
|
| 322 |
+
sessions_seen=len(
|
| 323 |
+
{
|
| 324 |
+
c.session_id
|
| 325 |
+
for c in rejected
|
| 326 |
+
if _extract_command_signature(c.failed_input.get("command", ""))
|
| 327 |
+
== cmd_sig
|
| 328 |
+
}
|
| 329 |
+
),
|
| 330 |
+
)
|
| 331 |
+
)
|
| 332 |
+
|
| 333 |
+
# Build failures: learn what command form works
|
| 334 |
+
build_fails = by_category.get(ErrorCategory.BUILD_FAILURE, [])
|
| 335 |
+
for c in build_fails:
|
| 336 |
+
failed_cmd = c.failed_input.get("command", "")
|
| 337 |
+
success_cmd = c.success_input.get("command", "")
|
| 338 |
+
if failed_cmd and success_cmd:
|
| 339 |
+
patterns.append(
|
| 340 |
+
CommandPattern(
|
| 341 |
+
category="build",
|
| 342 |
+
wrong_pattern=_extract_command_signature(failed_cmd),
|
| 343 |
+
correct_pattern=_extract_command_signature(success_cmd),
|
| 344 |
+
explanation="Build failed with first form, succeeded with second",
|
| 345 |
+
evidence_count=1,
|
| 346 |
+
)
|
| 347 |
+
)
|
| 348 |
+
|
| 349 |
+
# Module not found: learn correct python invocation
|
| 350 |
+
module_fails = by_category.get(ErrorCategory.MODULE_NOT_FOUND, [])
|
| 351 |
+
if module_fails:
|
| 352 |
+
wrong_pythons: Counter[str] = Counter()
|
| 353 |
+
correct_pythons: Counter[str] = Counter()
|
| 354 |
+
for c in module_fails:
|
| 355 |
+
wp = _extract_python_command(c.failed_input.get("command", ""))
|
| 356 |
+
cp = _extract_python_command(c.success_input.get("command", ""))
|
| 357 |
+
if wp:
|
| 358 |
+
wrong_pythons[wp] += 1
|
| 359 |
+
if cp:
|
| 360 |
+
correct_pythons[cp] += 1
|
| 361 |
+
|
| 362 |
+
if wrong_pythons and correct_pythons:
|
| 363 |
+
wrong = wrong_pythons.most_common(1)[0][0]
|
| 364 |
+
correct = correct_pythons.most_common(1)[0][0]
|
| 365 |
+
if wrong != correct:
|
| 366 |
+
patterns.append(
|
| 367 |
+
CommandPattern(
|
| 368 |
+
category="python_runtime",
|
| 369 |
+
wrong_pattern=f"`{wrong}` (modules not available)",
|
| 370 |
+
correct_pattern=f"`{correct}` (has project dependencies)",
|
| 371 |
+
explanation=f"Using `{wrong}` causes ModuleNotFoundError — use `{correct}` which has the project's venv",
|
| 372 |
+
evidence_count=sum(wrong_pythons.values()),
|
| 373 |
+
)
|
| 374 |
+
)
|
| 375 |
+
|
| 376 |
+
# Deduplicate patterns
|
| 377 |
+
seen = set()
|
| 378 |
+
unique = []
|
| 379 |
+
for p in patterns:
|
| 380 |
+
key = (p.category, p.wrong_pattern[:50])
|
| 381 |
+
if key not in seen:
|
| 382 |
+
seen.add(key)
|
| 383 |
+
unique.append(p)
|
| 384 |
+
return unique
|
| 385 |
+
|
| 386 |
+
|
| 387 |
+
# =============================================================================
|
| 388 |
+
# Retry Analyzer (uses corrections to provide specific suggestions)
|
| 389 |
+
# =============================================================================
|
| 390 |
+
|
| 391 |
+
|
| 392 |
+
def _analyze_retries(
|
| 393 |
+
sessions: list[SessionData], corrections: list[Correction]
|
| 394 |
+
) -> list[RetryPattern]:
|
| 395 |
+
"""Find stubborn retries with specific fix suggestions from corrections."""
|
| 396 |
+
patterns: list[RetryPattern] = []
|
| 397 |
+
|
| 398 |
+
# Build a correction lookup: (tool, error_category) → list of corrections
|
| 399 |
+
correction_lookup: dict[tuple[str, str], list[Correction]] = defaultdict(list)
|
| 400 |
+
for c in corrections:
|
| 401 |
+
correction_lookup[(c.tool_name, c.error_category.value)].append(c)
|
| 402 |
+
|
| 403 |
+
# Find retry streaks
|
| 404 |
+
pattern_counter: Counter[tuple[str, str, str]] = Counter()
|
| 405 |
+
max_retries: dict[tuple[str, str, str], int] = {}
|
| 406 |
+
|
| 407 |
+
for session in sessions:
|
| 408 |
+
streak: dict[str, list[ToolCall]] = defaultdict(list)
|
| 409 |
+
for tc in session.tool_calls:
|
| 410 |
+
key = f"{tc.name}:{tc.error_category.value}"
|
| 411 |
+
if tc.is_error:
|
| 412 |
+
streak[key].append(tc)
|
| 413 |
+
else:
|
| 414 |
+
if len(streak.get(key, [])) >= 3:
|
| 415 |
+
calls = streak[key]
|
| 416 |
+
pk = (tc.name, calls[0].error_category.value, calls[0].input_summary[:50])
|
| 417 |
+
pattern_counter[pk] += 1
|
| 418 |
+
max_retries[pk] = max(max_retries.get(pk, 0), len(calls))
|
| 419 |
+
streak[key] = []
|
| 420 |
+
for _key, calls in streak.items():
|
| 421 |
+
if len(calls) >= 3:
|
| 422 |
+
pk = (calls[0].name, calls[0].error_category.value, calls[0].input_summary[:50])
|
| 423 |
+
pattern_counter[pk] += 1
|
| 424 |
+
max_retries[pk] = max(max_retries.get(pk, 0), len(calls))
|
| 425 |
+
|
| 426 |
+
for (tool, err_cat, input_key), count in pattern_counter.most_common(10):
|
| 427 |
+
max_r = max_retries.get((tool, err_cat, input_key), 3)
|
| 428 |
+
|
| 429 |
+
# Try to get a SPECIFIC suggestion from corrections
|
| 430 |
+
relevant_corrections = correction_lookup.get((tool, err_cat), [])
|
| 431 |
+
suggestion = _build_specific_suggestion(tool, err_cat, relevant_corrections)
|
| 432 |
+
|
| 433 |
+
patterns.append(
|
| 434 |
+
RetryPattern(
|
| 435 |
+
tool_name=tool,
|
| 436 |
+
error_category=ErrorCategory(err_cat),
|
| 437 |
+
description=f"{tool} failing with {err_cat}: {input_key}",
|
| 438 |
+
max_retries_seen=max_r,
|
| 439 |
+
suggestion=suggestion,
|
| 440 |
+
evidence_count=count,
|
| 441 |
+
)
|
| 442 |
+
)
|
| 443 |
+
|
| 444 |
+
return patterns
|
| 445 |
+
|
| 446 |
+
|
| 447 |
+
def _build_specific_suggestion(
|
| 448 |
+
tool: str, error_category: str, corrections: list[Correction]
|
| 449 |
+
) -> str:
|
| 450 |
+
"""Build a specific suggestion from actual corrections, not generic advice."""
|
| 451 |
+
if not corrections:
|
| 452 |
+
# No corrections available — use tool+error specific defaults
|
| 453 |
+
return _default_suggestion(tool, error_category)
|
| 454 |
+
|
| 455 |
+
# Summarize what corrections tell us
|
| 456 |
+
if tool in ("Read", "read") and error_category == "file_not_found":
|
| 457 |
+
examples = []
|
| 458 |
+
for c in corrections[:3]:
|
| 459 |
+
wrong = _shorten_path(c.failed_input.get("file_path", ""))
|
| 460 |
+
right = _shorten_path(c.success_input.get("file_path", ""))
|
| 461 |
+
if wrong and right:
|
| 462 |
+
examples.append(f"`{wrong}` → `{right}`")
|
| 463 |
+
if examples:
|
| 464 |
+
return "Use Glob to discover actual path. Known corrections: " + "; ".join(examples)
|
| 465 |
+
|
| 466 |
+
if tool in ("Grep", "grep"):
|
| 467 |
+
# Summarize scope corrections
|
| 468 |
+
scopes = set()
|
| 469 |
+
for c in corrections[:5]:
|
| 470 |
+
right_path = c.success_input.get("path", "")
|
| 471 |
+
if right_path:
|
| 472 |
+
scopes.add(_shorten_path(right_path))
|
| 473 |
+
if scopes:
|
| 474 |
+
return f"Scope searches to: {', '.join(sorted(scopes)[:3])}"
|
| 475 |
+
|
| 476 |
+
if tool in ("Bash", "bash") and error_category == "user_rejected":
|
| 477 |
+
return "User prefers to run this command themselves. Show the command, don't execute it."
|
| 478 |
+
|
| 479 |
+
if tool in ("Bash", "bash") and error_category == "module_not_found":
|
| 480 |
+
correct_cmds = set()
|
| 481 |
+
for c in corrections[:5]:
|
| 482 |
+
prefix = _extract_python_command(c.success_input.get("command", ""))
|
| 483 |
+
if prefix:
|
| 484 |
+
correct_cmds.add(prefix)
|
| 485 |
+
if correct_cmds:
|
| 486 |
+
return f"Use {' or '.join(sorted(correct_cmds))} (has project dependencies)"
|
| 487 |
+
|
| 488 |
+
# Fallback: show one correction example
|
| 489 |
+
c = corrections[0]
|
| 490 |
+
return f"What worked: {c.success_summary[:80]}"
|
| 491 |
+
|
| 492 |
+
|
| 493 |
+
def _default_suggestion(tool: str, error_category: str) -> str:
|
| 494 |
+
"""Fallback when no corrections are available."""
|
| 495 |
+
defaults = {
|
| 496 |
+
(
|
| 497 |
+
"Glob",
|
| 498 |
+
"no_matches",
|
| 499 |
+
): "Broaden pattern to **/*.ext or use ls to explore directory structure",
|
| 500 |
+
("Grep", "no_matches"): "Try case-insensitive (-i) or broaden search scope",
|
| 501 |
+
("Grep", "timeout"): "Scope Grep to a specific subdirectory — the full repo is too large",
|
| 502 |
+
("Read", "file_not_found"): "Use Glob to discover the file path before Read",
|
| 503 |
+
("Read", "file_too_large"): "Use offset/limit parameters for this file",
|
| 504 |
+
("Bash", "module_not_found"): "Use the project's virtualenv Python",
|
| 505 |
+
("Bash", "permission_denied"): "Do not retry — try a different approach",
|
| 506 |
+
("Bash", "command_not_found"): "Verify tool is installed: which <tool>",
|
| 507 |
+
("Bash", "user_rejected"): "User does not want this command executed. Show it instead.",
|
| 508 |
+
("Edit", "unknown"): "If old_string has multiple matches, add more surrounding context",
|
| 509 |
+
}
|
| 510 |
+
return defaults.get((tool, error_category), "Try an alternative approach after 2 failures")
|
| 511 |
+
|
| 512 |
+
|
| 513 |
+
# =============================================================================
|
| 514 |
+
# Permission Analyzer
|
| 515 |
+
# =============================================================================
|
| 516 |
+
|
| 517 |
+
|
| 518 |
+
def _analyze_permissions(sessions: list[SessionData]) -> list[str]:
|
| 519 |
+
"""Find commands repeatedly denied — with specific advice."""
|
| 520 |
+
denied: Counter[str] = Counter()
|
| 521 |
+
denied_cmds: dict[str, str] = {} # key → full command example
|
| 522 |
+
|
| 523 |
+
for session in sessions:
|
| 524 |
+
for tc in session.tool_calls:
|
| 525 |
+
if not tc.is_error:
|
| 526 |
+
continue
|
| 527 |
+
if tc.error_category not in (
|
| 528 |
+
ErrorCategory.PERMISSION_DENIED,
|
| 529 |
+
ErrorCategory.USER_REJECTED,
|
| 530 |
+
):
|
| 531 |
+
continue
|
| 532 |
+
sig = _extract_command_signature(
|
| 533 |
+
tc.input_data.get("command", "")
|
| 534 |
+
if tc.name in ("Bash", "bash")
|
| 535 |
+
else tc.input_summary
|
| 536 |
+
)
|
| 537 |
+
key = f"{tc.name}: {sig}"
|
| 538 |
+
denied[key] += 1
|
| 539 |
+
if key not in denied_cmds:
|
| 540 |
+
denied_cmds[key] = tc.input_summary[:80]
|
| 541 |
+
|
| 542 |
+
results = []
|
| 543 |
+
for key, count in denied.most_common(10):
|
| 544 |
+
if count < 3:
|
| 545 |
+
break
|
| 546 |
+
results.append(
|
| 547 |
+
f"{key} — denied {count} times. Show the command to the user instead of executing it."
|
| 548 |
+
)
|
| 549 |
+
return results
|
| 550 |
+
|
| 551 |
+
|
| 552 |
+
# =============================================================================
|
| 553 |
+
# Cross-Session Analyzer
|
| 554 |
+
# =============================================================================
|
| 555 |
+
|
| 556 |
+
|
| 557 |
+
def _analyze_cross_session(sessions: list[SessionData]) -> list[str]:
|
| 558 |
+
"""Find failure patterns that repeat across 3+ sessions."""
|
| 559 |
+
pattern_sessions: dict[str, set[str]] = defaultdict(set)
|
| 560 |
+
|
| 561 |
+
for session in sessions:
|
| 562 |
+
for tc in session.tool_calls:
|
| 563 |
+
if not tc.is_error or tc.error_category == ErrorCategory.SIBLING_ERROR:
|
| 564 |
+
continue
|
| 565 |
+
key = f"{tc.name}|{tc.error_category.value}|{tc.input_summary[:60]}"
|
| 566 |
+
pattern_sessions[key].add(session.session_id)
|
| 567 |
+
|
| 568 |
+
cross_session = []
|
| 569 |
+
for key, session_ids in sorted(pattern_sessions.items(), key=lambda x: -len(x[1])):
|
| 570 |
+
if len(session_ids) < 3:
|
| 571 |
+
continue
|
| 572 |
+
parts = key.split("|", 2)
|
| 573 |
+
tool, err, inp = parts[0], parts[1], parts[2] if len(parts) > 2 else "?"
|
| 574 |
+
cross_session.append(f"{tool} {err}: {inp} (across {len(session_ids)} sessions)")
|
| 575 |
+
if len(cross_session) >= 15:
|
| 576 |
+
break
|
| 577 |
+
|
| 578 |
+
return cross_session
|
| 579 |
+
|
| 580 |
+
|
| 581 |
+
# =============================================================================
|
| 582 |
+
# Helpers
|
| 583 |
+
# =============================================================================
|
| 584 |
+
|
| 585 |
+
|
| 586 |
+
_PYTHON_CMD_RE = re.compile(
|
| 587 |
+
r"^((?:source\s+\S+\s*&&\s*)?(?:\.venv/bin/)?(?:python3?|uv run python|uv run|/opt/nflx/python))"
|
| 588 |
+
)
|
| 589 |
+
|
| 590 |
+
|
| 591 |
+
def _extract_python_command(cmd: str) -> str | None:
|
| 592 |
+
"""Extract the python invocation prefix from a command."""
|
| 593 |
+
cmd = cmd.strip()
|
| 594 |
+
if "&&" in cmd:
|
| 595 |
+
for part in cmd.split("&&"):
|
| 596 |
+
result = _extract_python_command(part.strip())
|
| 597 |
+
if result:
|
| 598 |
+
return result
|
| 599 |
+
return None
|
| 600 |
+
m = _PYTHON_CMD_RE.match(cmd)
|
| 601 |
+
return m.group(1) if m else None
|
| 602 |
+
|
| 603 |
+
|
| 604 |
+
def _extract_command_signature(cmd: str) -> str:
|
| 605 |
+
"""Extract a normalizable command signature (first ~60 chars, no args)."""
|
| 606 |
+
cmd = cmd.strip()
|
| 607 |
+
# Truncate at first newline
|
| 608 |
+
if "\n" in cmd:
|
| 609 |
+
cmd = cmd.split("\n")[0]
|
| 610 |
+
return cmd[:60]
|
| 611 |
+
|
| 612 |
+
|
| 613 |
+
def _shorten_path(path: str) -> str:
|
| 614 |
+
"""Make a path relative to home for readability."""
|
| 615 |
+
home = os.path.expanduser("~")
|
| 616 |
+
if path.startswith(home):
|
| 617 |
+
return "~" + path[len(home) :]
|
| 618 |
+
return path
|
headroom/learn/models.py
ADDED
|
@@ -0,0 +1,240 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Data models for Headroom Learn — tool-agnostic abstractions.
|
| 2 |
+
|
| 3 |
+
These models normalize tool call data from ANY agent system (Claude Code, Cursor,
|
| 4 |
+
Codex, custom agents) into a common format that analyzers can work with.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
from dataclasses import dataclass, field
|
| 10 |
+
from datetime import datetime
|
| 11 |
+
from enum import Enum
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
# =============================================================================
|
| 15 |
+
# Error Classification
|
| 16 |
+
# =============================================================================
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
class ErrorCategory(str, Enum):
|
| 20 |
+
"""Classified error categories for tool call failures."""
|
| 21 |
+
|
| 22 |
+
FILE_NOT_FOUND = "file_not_found"
|
| 23 |
+
MODULE_NOT_FOUND = "module_not_found"
|
| 24 |
+
COMMAND_NOT_FOUND = "command_not_found"
|
| 25 |
+
PERMISSION_DENIED = "permission_denied"
|
| 26 |
+
FILE_TOO_LARGE = "file_too_large"
|
| 27 |
+
IS_DIRECTORY = "is_directory"
|
| 28 |
+
SYNTAX_ERROR = "syntax_error"
|
| 29 |
+
RUNTIME_ERROR = "runtime_error"
|
| 30 |
+
TIMEOUT = "timeout"
|
| 31 |
+
NO_MATCHES = "no_matches" # Grep/Glob found nothing
|
| 32 |
+
USER_REJECTED = "user_rejected"
|
| 33 |
+
SIBLING_ERROR = "sibling_error" # Cascade from parallel call failure
|
| 34 |
+
EXIT_CODE = "exit_code"
|
| 35 |
+
CONNECTION_ERROR = "connection_error"
|
| 36 |
+
BUILD_FAILURE = "build_failure"
|
| 37 |
+
UNKNOWN = "unknown"
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
# =============================================================================
|
| 41 |
+
# Core Data Models (Tool-Agnostic)
|
| 42 |
+
# =============================================================================
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
@dataclass
|
| 46 |
+
class ToolCall:
|
| 47 |
+
"""A single tool call and its result — normalized from any agent system.
|
| 48 |
+
|
| 49 |
+
This is the fundamental unit of analysis. Scanners produce these,
|
| 50 |
+
analyzers consume them.
|
| 51 |
+
"""
|
| 52 |
+
|
| 53 |
+
name: str # Tool name ("Bash", "Read", "file_search", etc.)
|
| 54 |
+
tool_call_id: str # Unique ID linking call to result
|
| 55 |
+
input_data: dict # Tool input parameters
|
| 56 |
+
output: str # Result content (may be error message)
|
| 57 |
+
is_error: bool # Whether the call failed
|
| 58 |
+
error_category: ErrorCategory = ErrorCategory.UNKNOWN
|
| 59 |
+
msg_index: int = 0 # Position in conversation
|
| 60 |
+
output_bytes: int = 0 # Size of output
|
| 61 |
+
|
| 62 |
+
@property
|
| 63 |
+
def input_summary(self) -> str:
|
| 64 |
+
"""Short summary of tool input for display."""
|
| 65 |
+
if self.name in ("Bash", "bash"):
|
| 66 |
+
cmd: str = self.input_data.get("command", "")
|
| 67 |
+
return cmd[:100] + "..." if len(cmd) > 100 else cmd
|
| 68 |
+
if self.name in ("Read", "read"):
|
| 69 |
+
return str(self.input_data.get("file_path", "?"))
|
| 70 |
+
if self.name in ("Grep", "grep"):
|
| 71 |
+
return str(self.input_data.get("pattern", "?"))
|
| 72 |
+
if self.name in ("Glob", "glob"):
|
| 73 |
+
return str(self.input_data.get("pattern", "?"))
|
| 74 |
+
if self.name in ("Edit", "edit", "Write", "write"):
|
| 75 |
+
return str(self.input_data.get("file_path", "?"))
|
| 76 |
+
return str(self.input_data)[:80]
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
@dataclass
|
| 80 |
+
class SessionData:
|
| 81 |
+
"""Normalized data from a single conversation session."""
|
| 82 |
+
|
| 83 |
+
session_id: str
|
| 84 |
+
tool_calls: list[ToolCall] = field(default_factory=list)
|
| 85 |
+
timestamp: datetime | None = None
|
| 86 |
+
|
| 87 |
+
@property
|
| 88 |
+
def failure_count(self) -> int:
|
| 89 |
+
return sum(1 for tc in self.tool_calls if tc.is_error)
|
| 90 |
+
|
| 91 |
+
@property
|
| 92 |
+
def failure_rate(self) -> float:
|
| 93 |
+
if not self.tool_calls:
|
| 94 |
+
return 0.0
|
| 95 |
+
return self.failure_count / len(self.tool_calls)
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
@dataclass
|
| 99 |
+
class ProjectInfo:
|
| 100 |
+
"""Information about a project discovered by a scanner."""
|
| 101 |
+
|
| 102 |
+
name: str # Human-readable project name
|
| 103 |
+
project_path: Path # Actual project directory
|
| 104 |
+
data_path: Path # Where conversation logs are stored
|
| 105 |
+
context_file: Path | None = None # CLAUDE.md / .cursorrules / AGENTS.md
|
| 106 |
+
memory_file: Path | None = None # MEMORY.md or equivalent
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
# =============================================================================
|
| 110 |
+
# Analysis Output Models
|
| 111 |
+
# =============================================================================
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
class RecommendationTarget(str, Enum):
|
| 115 |
+
"""Where a recommendation should be written."""
|
| 116 |
+
|
| 117 |
+
CONTEXT_FILE = "context_file" # CLAUDE.md, .cursorrules, AGENTS.md
|
| 118 |
+
MEMORY_FILE = "memory_file" # MEMORY.md or equivalent
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
@dataclass
|
| 122 |
+
class EnvironmentFact:
|
| 123 |
+
"""A learned fact about the project's runtime environment."""
|
| 124 |
+
|
| 125 |
+
category: str # "python", "build_tool", "test_runner", "linter"
|
| 126 |
+
correct_command: str # What works: "uv run python"
|
| 127 |
+
wrong_commands: list[str] = field(default_factory=list) # What fails: ["python3"]
|
| 128 |
+
evidence_count: int = 0 # How many failures support this
|
| 129 |
+
sessions_seen: int = 0 # Across how many sessions
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
@dataclass
|
| 133 |
+
class StructureNote:
|
| 134 |
+
"""A learned fact about the project's file structure."""
|
| 135 |
+
|
| 136 |
+
category: str # "large_file", "missing_path", "path_correction", "search_scope"
|
| 137 |
+
path: str # The file path in question
|
| 138 |
+
note: str # Human-readable note
|
| 139 |
+
correct_path: str = "" # If corrected, what the actual path is
|
| 140 |
+
evidence_count: int = 0
|
| 141 |
+
sessions_seen: int = 0
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
@dataclass
|
| 145 |
+
class Correction:
|
| 146 |
+
"""A failure→success pair: what failed and what worked instead.
|
| 147 |
+
|
| 148 |
+
This is the core learning primitive. By comparing the failed input to
|
| 149 |
+
the successful input, we extract specific actionable knowledge.
|
| 150 |
+
"""
|
| 151 |
+
|
| 152 |
+
tool_name: str
|
| 153 |
+
failed_input: dict # The input that failed
|
| 154 |
+
success_input: dict # The input that succeeded
|
| 155 |
+
error_category: ErrorCategory
|
| 156 |
+
session_id: str
|
| 157 |
+
|
| 158 |
+
@property
|
| 159 |
+
def failed_summary(self) -> str:
|
| 160 |
+
if self.tool_name in ("Read", "read"):
|
| 161 |
+
return str(self.failed_input.get("file_path", "?"))
|
| 162 |
+
if self.tool_name in ("Bash", "bash"):
|
| 163 |
+
return str(self.failed_input.get("command", "?"))[:100]
|
| 164 |
+
if self.tool_name in ("Grep", "grep"):
|
| 165 |
+
path = str(self.failed_input.get("path", ""))
|
| 166 |
+
pattern = str(self.failed_input.get("pattern", ""))
|
| 167 |
+
return f"pattern={pattern[:40]} path={path}"
|
| 168 |
+
return str(self.failed_input)[:80]
|
| 169 |
+
|
| 170 |
+
@property
|
| 171 |
+
def success_summary(self) -> str:
|
| 172 |
+
if self.tool_name in ("Read", "read"):
|
| 173 |
+
return str(self.success_input.get("file_path", "?"))
|
| 174 |
+
if self.tool_name in ("Bash", "bash"):
|
| 175 |
+
return str(self.success_input.get("command", "?"))[:100]
|
| 176 |
+
if self.tool_name in ("Grep", "grep"):
|
| 177 |
+
path = str(self.success_input.get("path", ""))
|
| 178 |
+
pattern = str(self.success_input.get("pattern", ""))
|
| 179 |
+
return f"pattern={pattern[:40]} path={path}"
|
| 180 |
+
return str(self.success_input)[:80]
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
@dataclass
|
| 184 |
+
class CommandPattern:
|
| 185 |
+
"""A learned pattern about how commands should be run in this project."""
|
| 186 |
+
|
| 187 |
+
category: str # "gradle", "python", "test", "build", "lint"
|
| 188 |
+
wrong_pattern: str # What fails (e.g., "cd /path && ./gradlew")
|
| 189 |
+
correct_pattern: str # What works (e.g., "../gradlew from axion/")
|
| 190 |
+
explanation: str # Why (e.g., "user rejects cd-based gradle, use relative path")
|
| 191 |
+
evidence_count: int = 0
|
| 192 |
+
sessions_seen: int = 0
|
| 193 |
+
|
| 194 |
+
|
| 195 |
+
@dataclass
|
| 196 |
+
class RetryPattern:
|
| 197 |
+
"""A pattern of stubborn retries that should be prevented."""
|
| 198 |
+
|
| 199 |
+
tool_name: str
|
| 200 |
+
error_category: ErrorCategory
|
| 201 |
+
description: str # What keeps failing
|
| 202 |
+
max_retries_seen: int # Worst case observed
|
| 203 |
+
suggestion: str # What to do instead (SPECIFIC, from success correlation)
|
| 204 |
+
evidence_count: int = 0
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
@dataclass
|
| 208 |
+
class Recommendation:
|
| 209 |
+
"""A concrete recommendation to write to a context/memory file."""
|
| 210 |
+
|
| 211 |
+
target: RecommendationTarget
|
| 212 |
+
section: str # Section heading (e.g., "Environment", "Known Large Files")
|
| 213 |
+
content: str # Markdown content for the section
|
| 214 |
+
confidence: float = 0.0 # 0-1, based on evidence strength
|
| 215 |
+
evidence_count: int = 0 # Number of failures supporting this
|
| 216 |
+
|
| 217 |
+
|
| 218 |
+
@dataclass
|
| 219 |
+
class AnalysisReport:
|
| 220 |
+
"""Complete output of failure analysis for a project."""
|
| 221 |
+
|
| 222 |
+
project: ProjectInfo
|
| 223 |
+
total_calls: int = 0
|
| 224 |
+
total_failures: int = 0
|
| 225 |
+
total_sessions: int = 0
|
| 226 |
+
waste_bytes: int = 0
|
| 227 |
+
|
| 228 |
+
environment_facts: list[EnvironmentFact] = field(default_factory=list)
|
| 229 |
+
structure_notes: list[StructureNote] = field(default_factory=list)
|
| 230 |
+
retry_patterns: list[RetryPattern] = field(default_factory=list)
|
| 231 |
+
command_patterns: list[CommandPattern] = field(default_factory=list)
|
| 232 |
+
corrections: list[Correction] = field(default_factory=list)
|
| 233 |
+
permission_issues: list[str] = field(default_factory=list)
|
| 234 |
+
cross_session_patterns: list[str] = field(default_factory=list)
|
| 235 |
+
|
| 236 |
+
@property
|
| 237 |
+
def failure_rate(self) -> float:
|
| 238 |
+
if not self.total_calls:
|
| 239 |
+
return 0.0
|
| 240 |
+
return self.total_failures / self.total_calls
|
headroom/learn/scanner.py
ADDED
|
@@ -0,0 +1,358 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Conversation scanners — read tool call logs from different agent systems.
|
| 2 |
+
|
| 3 |
+
Scanners normalize conversation data into ToolCall sequences that analyzers
|
| 4 |
+
can process regardless of the source system.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import json
|
| 10 |
+
import logging
|
| 11 |
+
import re
|
| 12 |
+
from abc import ABC, abstractmethod
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
|
| 15 |
+
from .models import (
|
| 16 |
+
ErrorCategory,
|
| 17 |
+
ProjectInfo,
|
| 18 |
+
SessionData,
|
| 19 |
+
ToolCall,
|
| 20 |
+
)
|
| 21 |
+
|
| 22 |
+
logger = logging.getLogger(__name__)
|
| 23 |
+
|
| 24 |
+
# =============================================================================
|
| 25 |
+
# Error Classification
|
| 26 |
+
# =============================================================================
|
| 27 |
+
|
| 28 |
+
# Patterns checked in order — first match wins
|
| 29 |
+
_ERROR_PATTERNS: list[tuple[re.Pattern, ErrorCategory]] = [
|
| 30 |
+
(
|
| 31 |
+
re.compile(r"No such file or directory|ENOENT|FileNotFoundError|does not exist", re.I),
|
| 32 |
+
ErrorCategory.FILE_NOT_FOUND,
|
| 33 |
+
),
|
| 34 |
+
(
|
| 35 |
+
re.compile(r"ModuleNotFoundError|ImportError|No module named", re.I),
|
| 36 |
+
ErrorCategory.MODULE_NOT_FOUND,
|
| 37 |
+
),
|
| 38 |
+
(re.compile(r"command not found", re.I), ErrorCategory.COMMAND_NOT_FOUND),
|
| 39 |
+
(
|
| 40 |
+
re.compile(r"Permission denied|EACCES|EPERM|auto-denied", re.I),
|
| 41 |
+
ErrorCategory.PERMISSION_DENIED,
|
| 42 |
+
),
|
| 43 |
+
(
|
| 44 |
+
re.compile(r"file is too large|too many lines|exceeds.*limit", re.I),
|
| 45 |
+
ErrorCategory.FILE_TOO_LARGE,
|
| 46 |
+
),
|
| 47 |
+
(re.compile(r"EISDIR|Is a directory", re.I), ErrorCategory.IS_DIRECTORY),
|
| 48 |
+
(re.compile(r"SyntaxError|IndentationError", re.I), ErrorCategory.SYNTAX_ERROR),
|
| 49 |
+
(re.compile(r"Traceback \(most recent|Exception:|Error:", re.I), ErrorCategory.RUNTIME_ERROR),
|
| 50 |
+
(re.compile(r"timed? ?out|TimeoutError|deadline exceeded", re.I), ErrorCategory.TIMEOUT),
|
| 51 |
+
(re.compile(r"No (?:matches|files|results) found|0 matches", re.I), ErrorCategory.NO_MATCHES),
|
| 52 |
+
(
|
| 53 |
+
re.compile(r"user.*reject|user.*denied|declined|didn't want to proceed", re.I),
|
| 54 |
+
ErrorCategory.USER_REJECTED,
|
| 55 |
+
),
|
| 56 |
+
(re.compile(r"[Ss]ibling tool call errored", re.I), ErrorCategory.SIBLING_ERROR),
|
| 57 |
+
(re.compile(r"exit code|non-zero|exited with", re.I), ErrorCategory.EXIT_CODE),
|
| 58 |
+
(
|
| 59 |
+
re.compile(r"ConnectionError|ConnectionRefused|ECONNREFUSED|network", re.I),
|
| 60 |
+
ErrorCategory.CONNECTION_ERROR,
|
| 61 |
+
),
|
| 62 |
+
(
|
| 63 |
+
re.compile(r"BUILD FAILED|compilation error|compile error", re.I),
|
| 64 |
+
ErrorCategory.BUILD_FAILURE,
|
| 65 |
+
),
|
| 66 |
+
]
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def classify_error(content: str) -> ErrorCategory:
|
| 70 |
+
"""Classify an error message into a category."""
|
| 71 |
+
for pattern, category in _ERROR_PATTERNS:
|
| 72 |
+
if pattern.search(content[:2000]): # Only check first 2KB
|
| 73 |
+
return category
|
| 74 |
+
return ErrorCategory.UNKNOWN
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def is_error_content(content: str) -> bool:
|
| 78 |
+
"""Heuristic: does this tool result look like an error?"""
|
| 79 |
+
if not content or len(content) < 10:
|
| 80 |
+
return False
|
| 81 |
+
# Check for common error indicators in first 1KB
|
| 82 |
+
snippet = content[:1000]
|
| 83 |
+
indicators = [
|
| 84 |
+
"Error:",
|
| 85 |
+
"error:",
|
| 86 |
+
"ENOENT",
|
| 87 |
+
"No such file",
|
| 88 |
+
"command not found",
|
| 89 |
+
"Permission denied",
|
| 90 |
+
"ModuleNotFoundError",
|
| 91 |
+
"Traceback (most recent",
|
| 92 |
+
"FAILED",
|
| 93 |
+
"EISDIR",
|
| 94 |
+
"auto-denied",
|
| 95 |
+
"Sibling tool call errored",
|
| 96 |
+
"timed out",
|
| 97 |
+
"exit code",
|
| 98 |
+
"FileNotFoundError",
|
| 99 |
+
]
|
| 100 |
+
return any(ind in snippet for ind in indicators)
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
# =============================================================================
|
| 104 |
+
# Abstract Scanner
|
| 105 |
+
# =============================================================================
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
class ConversationScanner(ABC):
|
| 109 |
+
"""Base class for scanning conversation logs from any agent system.
|
| 110 |
+
|
| 111 |
+
Subclasses implement log format parsing for specific tools (Claude Code,
|
| 112 |
+
Cursor, Codex, etc.) and produce normalized ToolCall sequences.
|
| 113 |
+
"""
|
| 114 |
+
|
| 115 |
+
@abstractmethod
|
| 116 |
+
def discover_projects(self) -> list[ProjectInfo]:
|
| 117 |
+
"""Discover all projects with conversation data."""
|
| 118 |
+
...
|
| 119 |
+
|
| 120 |
+
@abstractmethod
|
| 121 |
+
def scan_project(self, project: ProjectInfo) -> list[SessionData]:
|
| 122 |
+
"""Scan all sessions for a project, returning normalized tool calls."""
|
| 123 |
+
...
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
# =============================================================================
|
| 127 |
+
# Claude Code Scanner
|
| 128 |
+
# =============================================================================
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
class ClaudeCodeScanner(ConversationScanner):
|
| 132 |
+
"""Reads Claude Code conversation logs from ~/.claude/projects/.
|
| 133 |
+
|
| 134 |
+
Claude Code stores conversations as JSONL files with these line types:
|
| 135 |
+
- type="assistant": message.content[] has tool_use blocks (name, input, id)
|
| 136 |
+
- type="user": message.content[] has tool_result blocks (tool_use_id, content)
|
| 137 |
+
"""
|
| 138 |
+
|
| 139 |
+
def __init__(self, claude_dir: Path | None = None):
|
| 140 |
+
self.claude_dir = claude_dir or Path.home() / ".claude"
|
| 141 |
+
self.projects_dir = self.claude_dir / "projects"
|
| 142 |
+
|
| 143 |
+
def discover_projects(self) -> list[ProjectInfo]:
|
| 144 |
+
"""Discover all projects under ~/.claude/projects/."""
|
| 145 |
+
if not self.projects_dir.exists():
|
| 146 |
+
return []
|
| 147 |
+
|
| 148 |
+
projects = []
|
| 149 |
+
for entry in sorted(self.projects_dir.iterdir()):
|
| 150 |
+
if not entry.is_dir() or entry.name.startswith("."):
|
| 151 |
+
continue
|
| 152 |
+
|
| 153 |
+
# Decode project path from escaped directory name
|
| 154 |
+
# e.g., "-Users-tchopra-claude-projects-headroom" → "/Users/tchopra/claude-projects/headroom"
|
| 155 |
+
project_path = Path("/" + entry.name.replace("-", "/", entry.name.count("-")))
|
| 156 |
+
|
| 157 |
+
# Try smarter decoding: split on segments that look like path components
|
| 158 |
+
# The escaping replaces / with - but also - in names stays as -
|
| 159 |
+
# Heuristic: try the decoded path, if it exists use it
|
| 160 |
+
decoded = _decode_project_path(entry.name)
|
| 161 |
+
if decoded:
|
| 162 |
+
project_path = decoded
|
| 163 |
+
|
| 164 |
+
# Derive human-readable name
|
| 165 |
+
name = project_path.name if project_path != Path("/") else entry.name
|
| 166 |
+
|
| 167 |
+
# Check for CLAUDE.md in actual project directory
|
| 168 |
+
context_file = None
|
| 169 |
+
if project_path.exists():
|
| 170 |
+
claude_md = project_path / "CLAUDE.md"
|
| 171 |
+
if claude_md.exists():
|
| 172 |
+
context_file = claude_md
|
| 173 |
+
|
| 174 |
+
# Check for MEMORY.md
|
| 175 |
+
memory_dir = entry / "memory"
|
| 176 |
+
memory_file = memory_dir / "MEMORY.md" if memory_dir.exists() else None
|
| 177 |
+
if memory_file and not memory_file.exists():
|
| 178 |
+
memory_file = None
|
| 179 |
+
|
| 180 |
+
# Only include projects with JSONL files
|
| 181 |
+
jsonl_files = list(entry.glob("*.jsonl"))
|
| 182 |
+
if not jsonl_files:
|
| 183 |
+
continue
|
| 184 |
+
|
| 185 |
+
projects.append(
|
| 186 |
+
ProjectInfo(
|
| 187 |
+
name=name,
|
| 188 |
+
project_path=project_path,
|
| 189 |
+
data_path=entry,
|
| 190 |
+
context_file=context_file,
|
| 191 |
+
memory_file=memory_file,
|
| 192 |
+
)
|
| 193 |
+
)
|
| 194 |
+
|
| 195 |
+
return projects
|
| 196 |
+
|
| 197 |
+
def scan_project(self, project: ProjectInfo) -> list[SessionData]:
|
| 198 |
+
"""Scan all conversation JSONL files for a project."""
|
| 199 |
+
sessions = []
|
| 200 |
+
|
| 201 |
+
# Find all JSONL files (main conversations, not subagent files)
|
| 202 |
+
jsonl_files = sorted(project.data_path.glob("*.jsonl"))
|
| 203 |
+
|
| 204 |
+
for jsonl_path in jsonl_files:
|
| 205 |
+
session = self._scan_session(jsonl_path)
|
| 206 |
+
if session and session.tool_calls:
|
| 207 |
+
sessions.append(session)
|
| 208 |
+
|
| 209 |
+
return sessions
|
| 210 |
+
|
| 211 |
+
def _scan_session(self, jsonl_path: Path) -> SessionData | None:
|
| 212 |
+
"""Scan a single JSONL conversation file."""
|
| 213 |
+
session_id = jsonl_path.stem
|
| 214 |
+
tool_uses: dict[str, tuple[str, dict]] = {} # tc_id → (tool_name, input)
|
| 215 |
+
tool_calls: list[ToolCall] = []
|
| 216 |
+
msg_index = 0
|
| 217 |
+
|
| 218 |
+
try:
|
| 219 |
+
with open(jsonl_path) as f:
|
| 220 |
+
for line in f:
|
| 221 |
+
try:
|
| 222 |
+
d = json.loads(line)
|
| 223 |
+
except json.JSONDecodeError:
|
| 224 |
+
continue
|
| 225 |
+
|
| 226 |
+
msg_index += 1
|
| 227 |
+
line_type = d.get("type", "")
|
| 228 |
+
|
| 229 |
+
if line_type == "assistant":
|
| 230 |
+
self._extract_tool_uses(d, tool_uses)
|
| 231 |
+
elif line_type == "user":
|
| 232 |
+
self._extract_tool_results(d, tool_uses, tool_calls, msg_index)
|
| 233 |
+
|
| 234 |
+
except (OSError, UnicodeDecodeError) as e:
|
| 235 |
+
logger.debug("Failed to read %s: %s", jsonl_path, e)
|
| 236 |
+
return None
|
| 237 |
+
|
| 238 |
+
return SessionData(session_id=session_id, tool_calls=tool_calls)
|
| 239 |
+
|
| 240 |
+
def _extract_tool_uses(self, d: dict, tool_uses: dict[str, tuple[str, dict]]) -> None:
|
| 241 |
+
"""Extract tool_use blocks from an assistant message."""
|
| 242 |
+
msg = d.get("message", {})
|
| 243 |
+
content = msg.get("content", [])
|
| 244 |
+
if not isinstance(content, list):
|
| 245 |
+
return
|
| 246 |
+
|
| 247 |
+
for block in content:
|
| 248 |
+
if not isinstance(block, dict) or block.get("type") != "tool_use":
|
| 249 |
+
continue
|
| 250 |
+
tc_id = block.get("id", "")
|
| 251 |
+
name = block.get("name", "")
|
| 252 |
+
inp = block.get("input", {})
|
| 253 |
+
if tc_id and name:
|
| 254 |
+
tool_uses[tc_id] = (name, inp if isinstance(inp, dict) else {})
|
| 255 |
+
|
| 256 |
+
def _extract_tool_results(
|
| 257 |
+
self,
|
| 258 |
+
d: dict,
|
| 259 |
+
tool_uses: dict[str, tuple[str, dict]],
|
| 260 |
+
tool_calls: list[ToolCall],
|
| 261 |
+
msg_index: int,
|
| 262 |
+
) -> None:
|
| 263 |
+
"""Extract tool_result blocks from a user message and match to tool_uses."""
|
| 264 |
+
msg = d.get("message", {})
|
| 265 |
+
content = msg.get("content", [])
|
| 266 |
+
if not isinstance(content, list):
|
| 267 |
+
return
|
| 268 |
+
|
| 269 |
+
for block in content:
|
| 270 |
+
if not isinstance(block, dict) or block.get("type") != "tool_result":
|
| 271 |
+
continue
|
| 272 |
+
|
| 273 |
+
tc_id = block.get("tool_use_id", "")
|
| 274 |
+
result_content = block.get("content", "")
|
| 275 |
+
if not isinstance(result_content, str):
|
| 276 |
+
result_content = str(result_content)
|
| 277 |
+
|
| 278 |
+
# Match to tool_use
|
| 279 |
+
if tc_id not in tool_uses:
|
| 280 |
+
continue
|
| 281 |
+
|
| 282 |
+
name, inp = tool_uses[tc_id]
|
| 283 |
+
|
| 284 |
+
# Determine if error
|
| 285 |
+
explicit_error = block.get("is_error", False)
|
| 286 |
+
detected_error = is_error_content(result_content)
|
| 287 |
+
is_err = explicit_error or detected_error
|
| 288 |
+
|
| 289 |
+
error_cat = classify_error(result_content) if is_err else ErrorCategory.UNKNOWN
|
| 290 |
+
|
| 291 |
+
tool_calls.append(
|
| 292 |
+
ToolCall(
|
| 293 |
+
name=name,
|
| 294 |
+
tool_call_id=tc_id,
|
| 295 |
+
input_data=inp,
|
| 296 |
+
output=result_content,
|
| 297 |
+
is_error=is_err,
|
| 298 |
+
error_category=error_cat,
|
| 299 |
+
msg_index=msg_index,
|
| 300 |
+
output_bytes=len(result_content.encode("utf-8")),
|
| 301 |
+
)
|
| 302 |
+
)
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
def _decode_project_path(escaped_name: str) -> Path | None:
|
| 306 |
+
"""Decode a Claude Code escaped project path.
|
| 307 |
+
|
| 308 |
+
Claude Code escapes paths by replacing / with -.
|
| 309 |
+
e.g., "-Users-tchopra-claude-projects-headroom"
|
| 310 |
+
→ "/Users/tchopra/claude-projects/headroom"
|
| 311 |
+
|
| 312 |
+
Since - is ambiguous (path separator vs literal hyphen), we try
|
| 313 |
+
progressively and check which decoded path actually exists.
|
| 314 |
+
"""
|
| 315 |
+
if not escaped_name.startswith("-"):
|
| 316 |
+
return None
|
| 317 |
+
|
| 318 |
+
# Simple approach: replace all - with / and check if path exists
|
| 319 |
+
simple = Path("/" + escaped_name[1:].replace("-", "/"))
|
| 320 |
+
if simple.exists():
|
| 321 |
+
return simple
|
| 322 |
+
|
| 323 |
+
# Try common patterns: /Users/username/...
|
| 324 |
+
parts = escaped_name[1:].split("-")
|
| 325 |
+
if len(parts) < 3:
|
| 326 |
+
return None
|
| 327 |
+
|
| 328 |
+
# Build path greedily: try joining with / and check existence
|
| 329 |
+
# Start with /Users/username (first 2 components are almost always correct)
|
| 330 |
+
if parts[0] == "Users" and len(parts) > 2:
|
| 331 |
+
base = Path(f"/{parts[0]}/{parts[1]}")
|
| 332 |
+
remaining = parts[2:]
|
| 333 |
+
return _greedy_path_decode(base, remaining)
|
| 334 |
+
|
| 335 |
+
return None
|
| 336 |
+
|
| 337 |
+
|
| 338 |
+
def _greedy_path_decode(base: Path, parts: list[str]) -> Path | None:
|
| 339 |
+
"""Greedily decode remaining path parts, trying - as / first."""
|
| 340 |
+
if not parts:
|
| 341 |
+
return base if base.exists() else None
|
| 342 |
+
|
| 343 |
+
# Try using / (this part is a directory component)
|
| 344 |
+
slash_path = base / parts[0]
|
| 345 |
+
result = _greedy_path_decode(slash_path, parts[1:])
|
| 346 |
+
if result:
|
| 347 |
+
return result
|
| 348 |
+
|
| 349 |
+
# Try joining with - (this part has a literal hyphen)
|
| 350 |
+
if len(parts) > 1:
|
| 351 |
+
hyphen_name = f"{parts[0]}-{parts[1]}"
|
| 352 |
+
hyphen_path = base / hyphen_name
|
| 353 |
+
result = _greedy_path_decode(hyphen_path, parts[2:])
|
| 354 |
+
if result:
|
| 355 |
+
return result
|
| 356 |
+
|
| 357 |
+
# If we've exhausted parts, check if current path exists
|
| 358 |
+
return base if base.exists() else None
|
headroom/learn/writer.py
ADDED
|
@@ -0,0 +1,333 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Context writers — write learned patterns to agent-specific context files.
|
| 2 |
+
|
| 3 |
+
Writers take Recommendations and write them to the appropriate context
|
| 4 |
+
injection mechanism for each agent system (CLAUDE.md, .cursorrules, etc.).
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import re
|
| 10 |
+
from abc import ABC, abstractmethod
|
| 11 |
+
from datetime import datetime, timezone
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
from .models import (
|
| 15 |
+
AnalysisReport,
|
| 16 |
+
CommandPattern,
|
| 17 |
+
EnvironmentFact,
|
| 18 |
+
ProjectInfo,
|
| 19 |
+
Recommendation,
|
| 20 |
+
RecommendationTarget,
|
| 21 |
+
RetryPattern,
|
| 22 |
+
StructureNote,
|
| 23 |
+
)
|
| 24 |
+
|
| 25 |
+
# Marker delimiters for Headroom-managed sections
|
| 26 |
+
_MARKER_START = "<!-- headroom:learn:start -->"
|
| 27 |
+
_MARKER_END = "<!-- headroom:learn:end -->"
|
| 28 |
+
_MARKER_PATTERN = re.compile(
|
| 29 |
+
re.escape(_MARKER_START) + r".*?" + re.escape(_MARKER_END),
|
| 30 |
+
re.DOTALL,
|
| 31 |
+
)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
# =============================================================================
|
| 35 |
+
# Recommender: AnalysisReport → Recommendations
|
| 36 |
+
# =============================================================================
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
class Recommender:
|
| 40 |
+
"""Converts an AnalysisReport into concrete markdown recommendations."""
|
| 41 |
+
|
| 42 |
+
def recommend(self, report: AnalysisReport) -> list[Recommendation]:
|
| 43 |
+
recommendations: list[Recommendation] = []
|
| 44 |
+
|
| 45 |
+
# Environment facts → CONTEXT_FILE (CLAUDE.md)
|
| 46 |
+
if report.environment_facts:
|
| 47 |
+
content = self._format_environment(report.environment_facts)
|
| 48 |
+
recommendations.append(
|
| 49 |
+
Recommendation(
|
| 50 |
+
target=RecommendationTarget.CONTEXT_FILE,
|
| 51 |
+
section="Environment",
|
| 52 |
+
content=content,
|
| 53 |
+
confidence=min(
|
| 54 |
+
1.0, sum(f.evidence_count for f in report.environment_facts) / 10
|
| 55 |
+
),
|
| 56 |
+
evidence_count=sum(f.evidence_count for f in report.environment_facts),
|
| 57 |
+
)
|
| 58 |
+
)
|
| 59 |
+
|
| 60 |
+
# Large files → CONTEXT_FILE
|
| 61 |
+
large_files = [n for n in report.structure_notes if n.category == "large_file"]
|
| 62 |
+
if large_files:
|
| 63 |
+
content = self._format_large_files(large_files)
|
| 64 |
+
recommendations.append(
|
| 65 |
+
Recommendation(
|
| 66 |
+
target=RecommendationTarget.CONTEXT_FILE,
|
| 67 |
+
section="Known Large Files",
|
| 68 |
+
content=content,
|
| 69 |
+
confidence=min(1.0, sum(n.evidence_count for n in large_files) / 5),
|
| 70 |
+
evidence_count=sum(n.evidence_count for n in large_files),
|
| 71 |
+
)
|
| 72 |
+
)
|
| 73 |
+
|
| 74 |
+
# Path corrections → CONTEXT_FILE (these are stable project structure facts)
|
| 75 |
+
path_corrections = [n for n in report.structure_notes if n.category == "path_correction"]
|
| 76 |
+
if path_corrections:
|
| 77 |
+
content = self._format_path_corrections(path_corrections)
|
| 78 |
+
recommendations.append(
|
| 79 |
+
Recommendation(
|
| 80 |
+
target=RecommendationTarget.CONTEXT_FILE,
|
| 81 |
+
section="File Path Corrections",
|
| 82 |
+
content=content,
|
| 83 |
+
confidence=0.9,
|
| 84 |
+
evidence_count=sum(n.evidence_count for n in path_corrections),
|
| 85 |
+
)
|
| 86 |
+
)
|
| 87 |
+
|
| 88 |
+
# Search scope corrections → CONTEXT_FILE
|
| 89 |
+
scope_corrections = [n for n in report.structure_notes if n.category == "search_scope"]
|
| 90 |
+
if scope_corrections:
|
| 91 |
+
content = self._format_scope_corrections(scope_corrections)
|
| 92 |
+
recommendations.append(
|
| 93 |
+
Recommendation(
|
| 94 |
+
target=RecommendationTarget.CONTEXT_FILE,
|
| 95 |
+
section="Search Scope",
|
| 96 |
+
content=content,
|
| 97 |
+
confidence=0.8,
|
| 98 |
+
evidence_count=sum(n.evidence_count for n in scope_corrections),
|
| 99 |
+
)
|
| 100 |
+
)
|
| 101 |
+
|
| 102 |
+
# Command patterns → CONTEXT_FILE (stable project-level facts)
|
| 103 |
+
if report.command_patterns:
|
| 104 |
+
content = self._format_command_patterns(report.command_patterns)
|
| 105 |
+
recommendations.append(
|
| 106 |
+
Recommendation(
|
| 107 |
+
target=RecommendationTarget.CONTEXT_FILE,
|
| 108 |
+
section="Command Patterns",
|
| 109 |
+
content=content,
|
| 110 |
+
confidence=0.9,
|
| 111 |
+
evidence_count=sum(p.evidence_count for p in report.command_patterns),
|
| 112 |
+
)
|
| 113 |
+
)
|
| 114 |
+
|
| 115 |
+
# Missing paths (no correction found) → MEMORY_FILE
|
| 116 |
+
missing_paths = [n for n in report.structure_notes if n.category == "missing_path"]
|
| 117 |
+
if missing_paths:
|
| 118 |
+
content = self._format_missing_paths(missing_paths)
|
| 119 |
+
recommendations.append(
|
| 120 |
+
Recommendation(
|
| 121 |
+
target=RecommendationTarget.MEMORY_FILE,
|
| 122 |
+
section="Known Missing Paths",
|
| 123 |
+
content=content,
|
| 124 |
+
confidence=0.6,
|
| 125 |
+
evidence_count=sum(n.evidence_count for n in missing_paths),
|
| 126 |
+
)
|
| 127 |
+
)
|
| 128 |
+
|
| 129 |
+
# Retry patterns (with specific suggestions) → MEMORY_FILE
|
| 130 |
+
if report.retry_patterns:
|
| 131 |
+
content = self._format_retry_patterns(report.retry_patterns)
|
| 132 |
+
recommendations.append(
|
| 133 |
+
Recommendation(
|
| 134 |
+
target=RecommendationTarget.MEMORY_FILE,
|
| 135 |
+
section="Retry Prevention",
|
| 136 |
+
content=content,
|
| 137 |
+
confidence=0.7,
|
| 138 |
+
evidence_count=sum(p.evidence_count for p in report.retry_patterns),
|
| 139 |
+
)
|
| 140 |
+
)
|
| 141 |
+
|
| 142 |
+
# Permission issues → MEMORY_FILE
|
| 143 |
+
if report.permission_issues:
|
| 144 |
+
content = self._format_permissions(report.permission_issues)
|
| 145 |
+
recommendations.append(
|
| 146 |
+
Recommendation(
|
| 147 |
+
target=RecommendationTarget.MEMORY_FILE,
|
| 148 |
+
section="Permission Notes",
|
| 149 |
+
content=content,
|
| 150 |
+
confidence=0.5,
|
| 151 |
+
evidence_count=len(report.permission_issues),
|
| 152 |
+
)
|
| 153 |
+
)
|
| 154 |
+
|
| 155 |
+
return recommendations
|
| 156 |
+
|
| 157 |
+
def _format_environment(self, facts: list[EnvironmentFact]) -> str:
|
| 158 |
+
lines = []
|
| 159 |
+
for fact in facts:
|
| 160 |
+
wrong = ", ".join(f"`{w}`" for w in fact.wrong_commands[:3])
|
| 161 |
+
lines.append(
|
| 162 |
+
f"- **{fact.category.title()}**: use `{fact.correct_command}` "
|
| 163 |
+
f"(not {wrong} — {fact.evidence_count} failures observed)"
|
| 164 |
+
)
|
| 165 |
+
return "\n".join(lines)
|
| 166 |
+
|
| 167 |
+
def _format_large_files(self, notes: list[StructureNote]) -> str:
|
| 168 |
+
lines = ["Always use `offset` and `limit` parameters with Read for these files:"]
|
| 169 |
+
for note in sorted(notes, key=lambda n: -n.evidence_count):
|
| 170 |
+
lines.append(f"- `{note.path}` ({note.note})")
|
| 171 |
+
return "\n".join(lines)
|
| 172 |
+
|
| 173 |
+
def _format_path_corrections(self, notes: list[StructureNote]) -> str:
|
| 174 |
+
lines = ["These file paths are commonly guessed wrong. Use the correct paths:"]
|
| 175 |
+
for note in sorted(notes, key=lambda n: -n.evidence_count):
|
| 176 |
+
lines.append(f"- `{note.path}` → actually at `{note.correct_path}`")
|
| 177 |
+
return "\n".join(lines)
|
| 178 |
+
|
| 179 |
+
def _format_scope_corrections(self, notes: list[StructureNote]) -> str:
|
| 180 |
+
lines = ["When searching, use these scopes (broader paths work, narrow ones fail):"]
|
| 181 |
+
for note in sorted(notes, key=lambda n: -n.evidence_count):
|
| 182 |
+
lines.append(f"- Don't search `{note.path}` → use `{note.correct_path}` instead")
|
| 183 |
+
return "\n".join(lines)
|
| 184 |
+
|
| 185 |
+
def _format_command_patterns(self, patterns: list[CommandPattern]) -> str:
|
| 186 |
+
lines = []
|
| 187 |
+
for p in sorted(patterns, key=lambda p: -p.evidence_count):
|
| 188 |
+
lines.append(f"- **{p.category}**: {p.explanation}")
|
| 189 |
+
lines.append(f" - Wrong: {p.wrong_pattern}")
|
| 190 |
+
lines.append(f" - Correct: {p.correct_pattern}")
|
| 191 |
+
return "\n".join(lines)
|
| 192 |
+
|
| 193 |
+
def _format_missing_paths(self, notes: list[StructureNote]) -> str:
|
| 194 |
+
lines = []
|
| 195 |
+
for note in sorted(notes, key=lambda n: -n.evidence_count):
|
| 196 |
+
lines.append(f"- `{note.path}` — {note.note}")
|
| 197 |
+
return "\n".join(lines)
|
| 198 |
+
|
| 199 |
+
def _format_retry_patterns(self, patterns: list[RetryPattern]) -> str:
|
| 200 |
+
lines = []
|
| 201 |
+
for p in sorted(patterns, key=lambda p: -p.evidence_count):
|
| 202 |
+
lines.append(f"- {p.description}")
|
| 203 |
+
lines.append(f" → {p.suggestion}")
|
| 204 |
+
return "\n".join(lines)
|
| 205 |
+
|
| 206 |
+
def _format_permissions(self, issues: list[str]) -> str:
|
| 207 |
+
lines = []
|
| 208 |
+
for issue in issues:
|
| 209 |
+
lines.append(f"- {issue}")
|
| 210 |
+
return "\n".join(lines)
|
| 211 |
+
|
| 212 |
+
|
| 213 |
+
# =============================================================================
|
| 214 |
+
# Abstract Writer
|
| 215 |
+
# =============================================================================
|
| 216 |
+
|
| 217 |
+
|
| 218 |
+
class ContextWriter(ABC):
|
| 219 |
+
"""Base class for writing recommendations to context/memory files."""
|
| 220 |
+
|
| 221 |
+
@abstractmethod
|
| 222 |
+
def write(
|
| 223 |
+
self,
|
| 224 |
+
recommendations: list[Recommendation],
|
| 225 |
+
project: ProjectInfo,
|
| 226 |
+
dry_run: bool = True,
|
| 227 |
+
) -> WriteResult: ...
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
# =============================================================================
|
| 231 |
+
# Write Result
|
| 232 |
+
# =============================================================================
|
| 233 |
+
|
| 234 |
+
|
| 235 |
+
class WriteResult:
|
| 236 |
+
"""Result of a write operation."""
|
| 237 |
+
|
| 238 |
+
def __init__(self) -> None:
|
| 239 |
+
self.files_written: list[Path] = []
|
| 240 |
+
self.content_by_file: dict[Path, str] = {}
|
| 241 |
+
self.dry_run: bool = True
|
| 242 |
+
|
| 243 |
+
def add(self, path: Path, content: str) -> None:
|
| 244 |
+
self.files_written.append(path)
|
| 245 |
+
self.content_by_file[path] = content
|
| 246 |
+
|
| 247 |
+
|
| 248 |
+
# =============================================================================
|
| 249 |
+
# Claude Code Writer
|
| 250 |
+
# =============================================================================
|
| 251 |
+
|
| 252 |
+
|
| 253 |
+
class ClaudeCodeWriter(ContextWriter):
|
| 254 |
+
"""Writes learned patterns to CLAUDE.md and MEMORY.md for Claude Code."""
|
| 255 |
+
|
| 256 |
+
def write(
|
| 257 |
+
self,
|
| 258 |
+
recommendations: list[Recommendation],
|
| 259 |
+
project: ProjectInfo,
|
| 260 |
+
dry_run: bool = True,
|
| 261 |
+
) -> WriteResult:
|
| 262 |
+
result = WriteResult()
|
| 263 |
+
result.dry_run = dry_run
|
| 264 |
+
|
| 265 |
+
# Group recommendations by target
|
| 266 |
+
context_recs = [r for r in recommendations if r.target == RecommendationTarget.CONTEXT_FILE]
|
| 267 |
+
memory_recs = [r for r in recommendations if r.target == RecommendationTarget.MEMORY_FILE]
|
| 268 |
+
|
| 269 |
+
# Generate CLAUDE.md content
|
| 270 |
+
if context_recs:
|
| 271 |
+
claude_md_path = self._resolve_context_path(project)
|
| 272 |
+
section_content = self._build_section(context_recs)
|
| 273 |
+
full_content = self._merge_into_file(claude_md_path, section_content)
|
| 274 |
+
result.add(claude_md_path, full_content)
|
| 275 |
+
|
| 276 |
+
if not dry_run:
|
| 277 |
+
claude_md_path.parent.mkdir(parents=True, exist_ok=True)
|
| 278 |
+
claude_md_path.write_text(full_content)
|
| 279 |
+
|
| 280 |
+
# Generate MEMORY.md content
|
| 281 |
+
if memory_recs:
|
| 282 |
+
memory_path = self._resolve_memory_path(project)
|
| 283 |
+
section_content = self._build_section(memory_recs)
|
| 284 |
+
full_content = self._merge_into_file(memory_path, section_content)
|
| 285 |
+
result.add(memory_path, full_content)
|
| 286 |
+
|
| 287 |
+
if not dry_run:
|
| 288 |
+
memory_path.parent.mkdir(parents=True, exist_ok=True)
|
| 289 |
+
memory_path.write_text(full_content)
|
| 290 |
+
|
| 291 |
+
return result
|
| 292 |
+
|
| 293 |
+
def _resolve_context_path(self, project: ProjectInfo) -> Path:
|
| 294 |
+
"""Resolve path for CLAUDE.md."""
|
| 295 |
+
if project.context_file:
|
| 296 |
+
return project.context_file
|
| 297 |
+
return project.project_path / "CLAUDE.md"
|
| 298 |
+
|
| 299 |
+
def _resolve_memory_path(self, project: ProjectInfo) -> Path:
|
| 300 |
+
"""Resolve path for MEMORY.md."""
|
| 301 |
+
if project.memory_file:
|
| 302 |
+
return project.memory_file
|
| 303 |
+
return project.data_path / "memory" / "MEMORY.md"
|
| 304 |
+
|
| 305 |
+
def _build_section(self, recommendations: list[Recommendation]) -> str:
|
| 306 |
+
"""Build the marker-delimited section content."""
|
| 307 |
+
now = datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
| 308 |
+
lines = [
|
| 309 |
+
_MARKER_START,
|
| 310 |
+
"## Headroom Learned Patterns",
|
| 311 |
+
f"*Auto-generated by `headroom learn` on {now} — do not edit manually*",
|
| 312 |
+
"",
|
| 313 |
+
]
|
| 314 |
+
|
| 315 |
+
for rec in recommendations:
|
| 316 |
+
lines.append(f"### {rec.section}")
|
| 317 |
+
lines.append(rec.content)
|
| 318 |
+
lines.append("")
|
| 319 |
+
|
| 320 |
+
lines.append(_MARKER_END)
|
| 321 |
+
return "\n".join(lines)
|
| 322 |
+
|
| 323 |
+
def _merge_into_file(self, file_path: Path, section: str) -> str:
|
| 324 |
+
"""Merge the section into an existing file, replacing any prior section."""
|
| 325 |
+
if file_path.exists():
|
| 326 |
+
existing = file_path.read_text()
|
| 327 |
+
# Replace existing headroom section
|
| 328 |
+
if _MARKER_START in existing:
|
| 329 |
+
return _MARKER_PATTERN.sub(section, existing)
|
| 330 |
+
# Append to end
|
| 331 |
+
return existing.rstrip() + "\n\n" + section + "\n"
|
| 332 |
+
else:
|
| 333 |
+
return section + "\n"
|
headroom/providers/anthropic.py
CHANGED
|
@@ -487,11 +487,11 @@ class AnthropicProvider(Provider):
|
|
| 487 |
info = litellm_get_model_info(model)
|
| 488 |
if info:
|
| 489 |
if "max_input_tokens" in info and info["max_input_tokens"] is not None:
|
| 490 |
-
limit = info["max_input_tokens"]
|
| 491 |
self._context_limits[model] = limit
|
| 492 |
return limit
|
| 493 |
if "max_tokens" in info and info["max_tokens"] is not None:
|
| 494 |
-
limit = info["max_tokens"]
|
| 495 |
self._context_limits[model] = limit
|
| 496 |
return limit
|
| 497 |
except Exception as e:
|
|
|
|
| 487 |
info = litellm_get_model_info(model)
|
| 488 |
if info:
|
| 489 |
if "max_input_tokens" in info and info["max_input_tokens"] is not None:
|
| 490 |
+
limit = int(info["max_input_tokens"])
|
| 491 |
self._context_limits[model] = limit
|
| 492 |
return limit
|
| 493 |
if "max_tokens" in info and info["max_tokens"] is not None:
|
| 494 |
+
limit = int(info["max_tokens"])
|
| 495 |
self._context_limits[model] = limit
|
| 496 |
return limit
|
| 497 |
except Exception as e:
|
tests/test_learn/__init__.py
ADDED
|
File without changes
|
tests/test_learn/test_analyzer.py
ADDED
|
@@ -0,0 +1,348 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tests for failure analyzers — generic tool call pattern recognition."""
|
| 2 |
+
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
|
| 5 |
+
from headroom.learn.analyzer import FailureAnalyzer
|
| 6 |
+
from headroom.learn.models import (
|
| 7 |
+
ErrorCategory,
|
| 8 |
+
ProjectInfo,
|
| 9 |
+
SessionData,
|
| 10 |
+
ToolCall,
|
| 11 |
+
)
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def _project() -> ProjectInfo:
|
| 15 |
+
return ProjectInfo(
|
| 16 |
+
name="test-project",
|
| 17 |
+
project_path=Path("/tmp/test-project"),
|
| 18 |
+
data_path=Path("/tmp/test-data"),
|
| 19 |
+
)
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _tc(
|
| 23 |
+
name: str = "Bash",
|
| 24 |
+
input_data: dict | None = None,
|
| 25 |
+
output: str = "ok",
|
| 26 |
+
is_error: bool = False,
|
| 27 |
+
error_category: ErrorCategory = ErrorCategory.UNKNOWN,
|
| 28 |
+
msg_index: int = 0,
|
| 29 |
+
) -> ToolCall:
|
| 30 |
+
return ToolCall(
|
| 31 |
+
name=name,
|
| 32 |
+
tool_call_id=f"tc_{msg_index}",
|
| 33 |
+
input_data=input_data or {},
|
| 34 |
+
output=output,
|
| 35 |
+
is_error=is_error,
|
| 36 |
+
error_category=error_category,
|
| 37 |
+
msg_index=msg_index,
|
| 38 |
+
output_bytes=len(output),
|
| 39 |
+
)
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
class TestAnalyzerBasics:
|
| 43 |
+
def test_empty_sessions(self):
|
| 44 |
+
analyzer = FailureAnalyzer()
|
| 45 |
+
report = analyzer.analyze(_project(), [])
|
| 46 |
+
assert report.total_calls == 0
|
| 47 |
+
assert report.total_failures == 0
|
| 48 |
+
assert report.failure_rate == 0.0
|
| 49 |
+
|
| 50 |
+
def test_no_failures(self):
|
| 51 |
+
analyzer = FailureAnalyzer()
|
| 52 |
+
sessions = [
|
| 53 |
+
SessionData(
|
| 54 |
+
session_id="s1",
|
| 55 |
+
tool_calls=[_tc(msg_index=i) for i in range(10)],
|
| 56 |
+
)
|
| 57 |
+
]
|
| 58 |
+
report = analyzer.analyze(_project(), sessions)
|
| 59 |
+
assert report.total_calls == 10
|
| 60 |
+
assert report.total_failures == 0
|
| 61 |
+
|
| 62 |
+
def test_basic_failure_counting(self):
|
| 63 |
+
analyzer = FailureAnalyzer()
|
| 64 |
+
sessions = [
|
| 65 |
+
SessionData(
|
| 66 |
+
session_id="s1",
|
| 67 |
+
tool_calls=[
|
| 68 |
+
_tc(msg_index=0),
|
| 69 |
+
_tc(msg_index=1, is_error=True, output="Error: something broke"),
|
| 70 |
+
_tc(msg_index=2),
|
| 71 |
+
],
|
| 72 |
+
)
|
| 73 |
+
]
|
| 74 |
+
report = analyzer.analyze(_project(), sessions)
|
| 75 |
+
assert report.total_calls == 3
|
| 76 |
+
assert report.total_failures == 1
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
class TestEnvironmentAnalyzer:
|
| 80 |
+
def test_detects_wrong_python(self):
|
| 81 |
+
"""Module not found with python3 + successes with uv run → learn correct command."""
|
| 82 |
+
analyzer = FailureAnalyzer()
|
| 83 |
+
sessions = [
|
| 84 |
+
SessionData(
|
| 85 |
+
session_id="s1",
|
| 86 |
+
tool_calls=[
|
| 87 |
+
# Failures with python3
|
| 88 |
+
_tc(
|
| 89 |
+
name="Bash",
|
| 90 |
+
input_data={"command": "python3 -c 'import mylib'"},
|
| 91 |
+
output="ModuleNotFoundError: No module named 'mylib'",
|
| 92 |
+
is_error=True,
|
| 93 |
+
error_category=ErrorCategory.MODULE_NOT_FOUND,
|
| 94 |
+
msg_index=0,
|
| 95 |
+
),
|
| 96 |
+
_tc(
|
| 97 |
+
name="Bash",
|
| 98 |
+
input_data={"command": "python3 -c 'import mylib'"},
|
| 99 |
+
output="ModuleNotFoundError",
|
| 100 |
+
is_error=True,
|
| 101 |
+
error_category=ErrorCategory.MODULE_NOT_FOUND,
|
| 102 |
+
msg_index=1,
|
| 103 |
+
),
|
| 104 |
+
# Success with uv run
|
| 105 |
+
_tc(
|
| 106 |
+
name="Bash",
|
| 107 |
+
input_data={"command": "uv run python -c 'import mylib'"},
|
| 108 |
+
output="ok",
|
| 109 |
+
msg_index=2,
|
| 110 |
+
),
|
| 111 |
+
_tc(
|
| 112 |
+
name="Bash",
|
| 113 |
+
input_data={"command": "uv run python -c 'import mylib'"},
|
| 114 |
+
output="ok",
|
| 115 |
+
msg_index=3,
|
| 116 |
+
),
|
| 117 |
+
_tc(
|
| 118 |
+
name="Bash",
|
| 119 |
+
input_data={"command": "uv run python -c 'import mylib'"},
|
| 120 |
+
output="ok",
|
| 121 |
+
msg_index=4,
|
| 122 |
+
),
|
| 123 |
+
],
|
| 124 |
+
)
|
| 125 |
+
]
|
| 126 |
+
report = analyzer.analyze(_project(), sessions)
|
| 127 |
+
assert len(report.environment_facts) >= 1
|
| 128 |
+
fact = report.environment_facts[0]
|
| 129 |
+
assert fact.category == "python"
|
| 130 |
+
assert "uv run" in fact.correct_command
|
| 131 |
+
assert "python3" in fact.wrong_commands
|
| 132 |
+
|
| 133 |
+
|
| 134 |
+
class TestStructureAnalyzer:
|
| 135 |
+
def test_detects_missing_paths(self):
|
| 136 |
+
"""Files that repeatedly fail Read → learned as missing."""
|
| 137 |
+
analyzer = FailureAnalyzer()
|
| 138 |
+
sessions = [
|
| 139 |
+
SessionData(
|
| 140 |
+
session_id="s1",
|
| 141 |
+
tool_calls=[
|
| 142 |
+
_tc(
|
| 143 |
+
name="Read",
|
| 144 |
+
input_data={"file_path": "/src/missing.py"},
|
| 145 |
+
output="No such file",
|
| 146 |
+
is_error=True,
|
| 147 |
+
error_category=ErrorCategory.FILE_NOT_FOUND,
|
| 148 |
+
msg_index=0,
|
| 149 |
+
),
|
| 150 |
+
],
|
| 151 |
+
),
|
| 152 |
+
SessionData(
|
| 153 |
+
session_id="s2",
|
| 154 |
+
tool_calls=[
|
| 155 |
+
_tc(
|
| 156 |
+
name="Read",
|
| 157 |
+
input_data={"file_path": "/src/missing.py"},
|
| 158 |
+
output="No such file",
|
| 159 |
+
is_error=True,
|
| 160 |
+
error_category=ErrorCategory.FILE_NOT_FOUND,
|
| 161 |
+
msg_index=0,
|
| 162 |
+
),
|
| 163 |
+
],
|
| 164 |
+
),
|
| 165 |
+
]
|
| 166 |
+
report = analyzer.analyze(_project(), sessions)
|
| 167 |
+
missing = [n for n in report.structure_notes if n.category == "missing_path"]
|
| 168 |
+
assert len(missing) >= 1
|
| 169 |
+
assert "/src/missing.py" in missing[0].path
|
| 170 |
+
|
| 171 |
+
def test_detects_large_files(self):
|
| 172 |
+
"""Files that repeatedly trigger too-large errors → learned."""
|
| 173 |
+
analyzer = FailureAnalyzer()
|
| 174 |
+
sessions = [
|
| 175 |
+
SessionData(
|
| 176 |
+
session_id="s1",
|
| 177 |
+
tool_calls=[
|
| 178 |
+
_tc(
|
| 179 |
+
name="Read",
|
| 180 |
+
input_data={"file_path": "/src/huge.py"},
|
| 181 |
+
output="file is too large",
|
| 182 |
+
is_error=True,
|
| 183 |
+
error_category=ErrorCategory.FILE_TOO_LARGE,
|
| 184 |
+
msg_index=0,
|
| 185 |
+
),
|
| 186 |
+
_tc(
|
| 187 |
+
name="Read",
|
| 188 |
+
input_data={"file_path": "/src/huge.py"},
|
| 189 |
+
output="file is too large",
|
| 190 |
+
is_error=True,
|
| 191 |
+
error_category=ErrorCategory.FILE_TOO_LARGE,
|
| 192 |
+
msg_index=1,
|
| 193 |
+
),
|
| 194 |
+
],
|
| 195 |
+
),
|
| 196 |
+
]
|
| 197 |
+
report = analyzer.analyze(_project(), sessions)
|
| 198 |
+
large = [n for n in report.structure_notes if n.category == "large_file"]
|
| 199 |
+
assert len(large) >= 1
|
| 200 |
+
assert "/src/huge.py" in large[0].path
|
| 201 |
+
|
| 202 |
+
def test_single_occurrence_not_reported(self):
|
| 203 |
+
"""A single file_not_found shouldn't be reported (might be transient)."""
|
| 204 |
+
analyzer = FailureAnalyzer()
|
| 205 |
+
sessions = [
|
| 206 |
+
SessionData(
|
| 207 |
+
session_id="s1",
|
| 208 |
+
tool_calls=[
|
| 209 |
+
_tc(
|
| 210 |
+
name="Read",
|
| 211 |
+
input_data={"file_path": "/src/one_time.py"},
|
| 212 |
+
output="No such file",
|
| 213 |
+
is_error=True,
|
| 214 |
+
error_category=ErrorCategory.FILE_NOT_FOUND,
|
| 215 |
+
msg_index=0,
|
| 216 |
+
),
|
| 217 |
+
],
|
| 218 |
+
)
|
| 219 |
+
]
|
| 220 |
+
report = analyzer.analyze(_project(), sessions)
|
| 221 |
+
missing = [n for n in report.structure_notes if n.category == "missing_path"]
|
| 222 |
+
assert len(missing) == 0
|
| 223 |
+
|
| 224 |
+
|
| 225 |
+
class TestRetryAnalyzer:
|
| 226 |
+
def test_detects_stubborn_retries(self):
|
| 227 |
+
"""Same tool failing 3+ times in a row → retry pattern."""
|
| 228 |
+
analyzer = FailureAnalyzer()
|
| 229 |
+
sessions = [
|
| 230 |
+
SessionData(
|
| 231 |
+
session_id="s1",
|
| 232 |
+
tool_calls=[
|
| 233 |
+
_tc(
|
| 234 |
+
name="Bash",
|
| 235 |
+
input_data={"command": "mkdir -p /x"},
|
| 236 |
+
output="auto-denied",
|
| 237 |
+
is_error=True,
|
| 238 |
+
error_category=ErrorCategory.PERMISSION_DENIED,
|
| 239 |
+
msg_index=i,
|
| 240 |
+
)
|
| 241 |
+
for i in range(5)
|
| 242 |
+
],
|
| 243 |
+
)
|
| 244 |
+
]
|
| 245 |
+
report = analyzer.analyze(_project(), sessions)
|
| 246 |
+
assert len(report.retry_patterns) >= 1
|
| 247 |
+
pattern = report.retry_patterns[0]
|
| 248 |
+
assert pattern.tool_name == "Bash"
|
| 249 |
+
assert pattern.max_retries_seen >= 5
|
| 250 |
+
|
| 251 |
+
def test_two_failures_not_stubborn(self):
|
| 252 |
+
"""Only 2 failures shouldn't trigger a retry pattern (threshold is 3)."""
|
| 253 |
+
analyzer = FailureAnalyzer()
|
| 254 |
+
sessions = [
|
| 255 |
+
SessionData(
|
| 256 |
+
session_id="s1",
|
| 257 |
+
tool_calls=[
|
| 258 |
+
_tc(
|
| 259 |
+
name="Glob",
|
| 260 |
+
output="No matches",
|
| 261 |
+
is_error=True,
|
| 262 |
+
error_category=ErrorCategory.NO_MATCHES,
|
| 263 |
+
msg_index=0,
|
| 264 |
+
),
|
| 265 |
+
_tc(
|
| 266 |
+
name="Glob",
|
| 267 |
+
output="No matches",
|
| 268 |
+
is_error=True,
|
| 269 |
+
error_category=ErrorCategory.NO_MATCHES,
|
| 270 |
+
msg_index=1,
|
| 271 |
+
),
|
| 272 |
+
_tc(name="Glob", output="found.py", msg_index=2), # Success breaks streak
|
| 273 |
+
],
|
| 274 |
+
)
|
| 275 |
+
]
|
| 276 |
+
report = analyzer.analyze(_project(), sessions)
|
| 277 |
+
assert len(report.retry_patterns) == 0
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
class TestCrossSessionAnalyzer:
|
| 281 |
+
def test_cross_session_pattern(self):
|
| 282 |
+
"""Same failure in 3+ sessions → cross-session pattern."""
|
| 283 |
+
analyzer = FailureAnalyzer()
|
| 284 |
+
sessions = [
|
| 285 |
+
SessionData(
|
| 286 |
+
session_id=f"s{i}",
|
| 287 |
+
tool_calls=[
|
| 288 |
+
_tc(
|
| 289 |
+
name="Read",
|
| 290 |
+
input_data={"file_path": "/docs/RESEARCH.md"},
|
| 291 |
+
output="No such file",
|
| 292 |
+
is_error=True,
|
| 293 |
+
error_category=ErrorCategory.FILE_NOT_FOUND,
|
| 294 |
+
msg_index=0,
|
| 295 |
+
),
|
| 296 |
+
],
|
| 297 |
+
)
|
| 298 |
+
for i in range(4)
|
| 299 |
+
]
|
| 300 |
+
report = analyzer.analyze(_project(), sessions)
|
| 301 |
+
assert len(report.cross_session_patterns) >= 1
|
| 302 |
+
assert any("RESEARCH.md" in p for p in report.cross_session_patterns)
|
| 303 |
+
|
| 304 |
+
def test_two_sessions_not_enough(self):
|
| 305 |
+
"""Only 2 sessions shouldn't trigger cross-session (threshold is 3)."""
|
| 306 |
+
analyzer = FailureAnalyzer()
|
| 307 |
+
sessions = [
|
| 308 |
+
SessionData(
|
| 309 |
+
session_id=f"s{i}",
|
| 310 |
+
tool_calls=[
|
| 311 |
+
_tc(
|
| 312 |
+
name="Read",
|
| 313 |
+
input_data={"file_path": "/rare.py"},
|
| 314 |
+
output="No such file",
|
| 315 |
+
is_error=True,
|
| 316 |
+
error_category=ErrorCategory.FILE_NOT_FOUND,
|
| 317 |
+
msg_index=0,
|
| 318 |
+
),
|
| 319 |
+
],
|
| 320 |
+
)
|
| 321 |
+
for i in range(2)
|
| 322 |
+
]
|
| 323 |
+
report = analyzer.analyze(_project(), sessions)
|
| 324 |
+
assert len(report.cross_session_patterns) == 0
|
| 325 |
+
|
| 326 |
+
|
| 327 |
+
class TestPermissionAnalyzer:
|
| 328 |
+
def test_detects_repeated_denials(self):
|
| 329 |
+
"""Commands denied 3+ times → permission note."""
|
| 330 |
+
analyzer = FailureAnalyzer()
|
| 331 |
+
sessions = [
|
| 332 |
+
SessionData(
|
| 333 |
+
session_id="s1",
|
| 334 |
+
tool_calls=[
|
| 335 |
+
_tc(
|
| 336 |
+
name="Bash",
|
| 337 |
+
input_data={"command": "mkdir -p /x"},
|
| 338 |
+
output="auto-denied",
|
| 339 |
+
is_error=True,
|
| 340 |
+
error_category=ErrorCategory.PERMISSION_DENIED,
|
| 341 |
+
msg_index=i,
|
| 342 |
+
)
|
| 343 |
+
for i in range(4)
|
| 344 |
+
],
|
| 345 |
+
)
|
| 346 |
+
]
|
| 347 |
+
report = analyzer.analyze(_project(), sessions)
|
| 348 |
+
assert len(report.permission_issues) >= 1
|
tests/test_learn/test_writer.py
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tests for recommendation writer — marker-based file updates."""
|
| 2 |
+
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
|
| 5 |
+
from headroom.learn.models import ProjectInfo, Recommendation, RecommendationTarget
|
| 6 |
+
from headroom.learn.writer import _MARKER_END, _MARKER_START, ClaudeCodeWriter
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def _project(tmp_path: Path) -> ProjectInfo:
|
| 10 |
+
proj_dir = tmp_path / "myproject"
|
| 11 |
+
proj_dir.mkdir()
|
| 12 |
+
data_dir = tmp_path / "data"
|
| 13 |
+
data_dir.mkdir()
|
| 14 |
+
memory_dir = data_dir / "memory"
|
| 15 |
+
memory_dir.mkdir()
|
| 16 |
+
return ProjectInfo(
|
| 17 |
+
name="myproject",
|
| 18 |
+
project_path=proj_dir,
|
| 19 |
+
data_path=data_dir,
|
| 20 |
+
)
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def _rec(target: RecommendationTarget, section: str, content: str) -> Recommendation:
|
| 24 |
+
return Recommendation(
|
| 25 |
+
target=target, section=section, content=content, confidence=0.8, evidence_count=5
|
| 26 |
+
)
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
class TestClaudeCodeWriter:
|
| 30 |
+
def test_dry_run_does_not_write(self, tmp_path):
|
| 31 |
+
proj = _project(tmp_path)
|
| 32 |
+
writer = ClaudeCodeWriter()
|
| 33 |
+
recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- Use uv")]
|
| 34 |
+
|
| 35 |
+
result = writer.write(recs, proj, dry_run=True)
|
| 36 |
+
|
| 37 |
+
assert result.dry_run is True
|
| 38 |
+
assert len(result.files_written) == 1
|
| 39 |
+
# File should NOT exist (dry run)
|
| 40 |
+
claude_md = proj.project_path / "CLAUDE.md"
|
| 41 |
+
assert not claude_md.exists()
|
| 42 |
+
|
| 43 |
+
def test_apply_writes_claude_md(self, tmp_path):
|
| 44 |
+
proj = _project(tmp_path)
|
| 45 |
+
writer = ClaudeCodeWriter()
|
| 46 |
+
recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- Use `uv run python`")]
|
| 47 |
+
|
| 48 |
+
result = writer.write(recs, proj, dry_run=False)
|
| 49 |
+
|
| 50 |
+
assert result.dry_run is False
|
| 51 |
+
claude_md = proj.project_path / "CLAUDE.md"
|
| 52 |
+
assert claude_md.exists()
|
| 53 |
+
content = claude_md.read_text()
|
| 54 |
+
assert "uv run python" in content
|
| 55 |
+
assert _MARKER_START in content
|
| 56 |
+
assert _MARKER_END in content
|
| 57 |
+
|
| 58 |
+
def test_apply_writes_memory_md(self, tmp_path):
|
| 59 |
+
proj = _project(tmp_path)
|
| 60 |
+
writer = ClaudeCodeWriter()
|
| 61 |
+
recs = [_rec(RecommendationTarget.MEMORY_FILE, "Retry Prevention", "- Don't retry globs")]
|
| 62 |
+
|
| 63 |
+
writer.write(recs, proj, dry_run=False)
|
| 64 |
+
|
| 65 |
+
memory_md = proj.data_path / "memory" / "MEMORY.md"
|
| 66 |
+
assert memory_md.exists()
|
| 67 |
+
assert "Don't retry globs" in memory_md.read_text()
|
| 68 |
+
|
| 69 |
+
def test_preserves_existing_claude_md_content(self, tmp_path):
|
| 70 |
+
proj = _project(tmp_path)
|
| 71 |
+
claude_md = proj.project_path / "CLAUDE.md"
|
| 72 |
+
claude_md.write_text("# My Project\n\nExisting instructions here.\n")
|
| 73 |
+
|
| 74 |
+
writer = ClaudeCodeWriter()
|
| 75 |
+
recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- Use uv")]
|
| 76 |
+
writer.write(recs, proj, dry_run=False)
|
| 77 |
+
|
| 78 |
+
content = claude_md.read_text()
|
| 79 |
+
assert "My Project" in content
|
| 80 |
+
assert "Existing instructions here" in content
|
| 81 |
+
assert "Use uv" in content
|
| 82 |
+
|
| 83 |
+
def test_replaces_existing_headroom_section(self, tmp_path):
|
| 84 |
+
proj = _project(tmp_path)
|
| 85 |
+
claude_md = proj.project_path / "CLAUDE.md"
|
| 86 |
+
old_section = (
|
| 87 |
+
f"# My Project\n\n{_MARKER_START}\n## Old Patterns\nold stuff\n{_MARKER_END}\n"
|
| 88 |
+
)
|
| 89 |
+
claude_md.write_text(old_section)
|
| 90 |
+
|
| 91 |
+
writer = ClaudeCodeWriter()
|
| 92 |
+
recs = [_rec(RecommendationTarget.CONTEXT_FILE, "Environment", "- New stuff")]
|
| 93 |
+
writer.write(recs, proj, dry_run=False)
|
| 94 |
+
|
| 95 |
+
content = claude_md.read_text()
|
| 96 |
+
assert "old stuff" not in content
|
| 97 |
+
assert "New stuff" in content
|
| 98 |
+
assert "My Project" in content
|
| 99 |
+
# Should have exactly one marker pair
|
| 100 |
+
assert content.count(_MARKER_START) == 1
|
| 101 |
+
assert content.count(_MARKER_END) == 1
|
| 102 |
+
|
| 103 |
+
def test_appends_to_existing_memory_md(self, tmp_path):
|
| 104 |
+
proj = _project(tmp_path)
|
| 105 |
+
memory_md = proj.data_path / "memory" / "MEMORY.md"
|
| 106 |
+
memory_md.write_text("# Existing Memory\n\nSome facts.\n")
|
| 107 |
+
|
| 108 |
+
writer = ClaudeCodeWriter()
|
| 109 |
+
recs = [_rec(RecommendationTarget.MEMORY_FILE, "Retry Prevention", "- New pattern")]
|
| 110 |
+
writer.write(recs, proj, dry_run=False)
|
| 111 |
+
|
| 112 |
+
content = memory_md.read_text()
|
| 113 |
+
assert "Existing Memory" in content
|
| 114 |
+
assert "Some facts" in content
|
| 115 |
+
assert "New pattern" in content
|