Gyeonghun Park Claude Sonnet 4.6 commited on
Commit
406a299
·
1 Parent(s): 4cc6fea

feat(learn): add CLI-based LLM backends for keyless headroom learn

Browse files

Allow `headroom learn` to use locally installed coding agent CLIs
(claude, gemini, codex) as LLM backends, so subscription users
without raw API keys can run failure analysis.

Priority: --model flag > API key > HEADROOM_LEARN_CLI env var > auto-detect

- Pass prompts via stdin to avoid ARG_MAX limits
- Handle TimeoutExpired, truncate stderr, enrich JSONDecodeError
- Add 31 new tests (48 total), all passing
- Update docs/learn.md with CLI backend documentation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs/learn.md CHANGED
@@ -145,9 +145,39 @@ Options:
145
  --project PATH Project directory to analyze (default: current directory)
146
  --all Analyze all discovered projects
147
  --apply Write recommendations (default: dry-run)
148
- --claude-dir PATH Path to .claude directory (default: ~/.claude)
 
149
  ```
150
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
  ## Real-World Results
152
 
153
  Tested on 67,583 tool calls across 23 projects:
 
145
  --project PATH Project directory to analyze (default: current directory)
146
  --all Analyze all discovered projects
147
  --apply Write recommendations (default: dry-run)
148
+ --model TEXT LLM model for analysis (default: auto-detected)
149
+ --agent TEXT Which coding agent to analyze: auto, claude, codex, gemini
150
  ```
151
 
152
+ ## LLM Backend Selection
153
+
154
+ `headroom learn` needs an LLM to analyze your sessions. It picks one automatically using this priority:
155
+
156
+ | Priority | Source | Example |
157
+ |----------|--------|---------|
158
+ | 1 | `--model` flag | `headroom learn --model gpt-4o` |
159
+ | 2 | API key env var | `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY` |
160
+ | 3 | `HEADROOM_LEARN_CLI` env var | `export HEADROOM_LEARN_CLI=gemini` |
161
+ | 4 | Auto-detect installed CLIs | Checks PATH for `claude`, `gemini`, `codex` |
162
+
163
+ ### Using without an API key
164
+
165
+ If you use Claude Code, Gemini CLI, or Codex via subscription (no raw API key), `headroom learn` can call them directly:
166
+
167
+ ```bash
168
+ # Auto-detects claude in PATH — no API key needed
169
+ headroom learn
170
+
171
+ # Explicitly select a CLI backend
172
+ headroom learn --model gemini-cli
173
+
174
+ # Pin a CLI via environment variable
175
+ export HEADROOM_LEARN_CLI=codex
176
+ headroom learn
177
+ ```
178
+
179
+ Valid values for `HEADROOM_LEARN_CLI`: `claude`, `gemini`, `codex`.
180
+
181
  ## Real-World Results
182
 
183
  Tested on 67,583 tool calls across 23 projects:
headroom/learn/analyzer.py CHANGED
@@ -8,6 +8,8 @@ structured recommendations for CLAUDE.md / MEMORY.md.
8
 
9
  Supports any LLM provider via LiteLLM: Anthropic, OpenAI, Google, Bedrock,
10
  Ollama, and 100+ others. Auto-detects the best available model from env vars.
 
 
11
  """
12
 
13
  from __future__ import annotations
@@ -15,6 +17,8 @@ from __future__ import annotations
15
  import json
16
  import logging
17
  import os
 
 
18
 
19
  from .models import (
20
  AnalysisResult,
@@ -37,17 +41,61 @@ _MODEL_DEFAULTS: list[tuple[str, str]] = [
37
 
38
  _MAX_DIGEST_TOKENS = 80_000 # Budget for the digest (leave room for prompt + output)
39
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
 
41
  def _detect_default_model() -> str:
42
- """Pick the best available model based on which API keys are set."""
 
 
 
 
 
 
 
 
43
  for env_var, model in _MODEL_DEFAULTS:
44
  if os.environ.get(env_var):
45
  return model
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
  raise RuntimeError(
47
  "No LLM API key found. headroom learn needs one of:\n"
48
  " export ANTHROPIC_API_KEY=sk-ant-... → uses claude-sonnet-4-6\n"
49
  " export OPENAI_API_KEY=sk-... → uses gpt-4o\n"
50
  " export GEMINI_API_KEY=... → uses gemini-2.0-flash\n"
 
 
51
  "Or specify a model directly: headroom learn --model <litellm-model-name>"
52
  )
53
 
@@ -263,12 +311,113 @@ Return ONLY valid JSON matching this schema — no other text:
263
  """
264
 
265
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
266
  def _call_llm(digest: str, model: str) -> dict:
267
  """Call LLM with the session digest and return parsed JSON.
268
 
269
  Uses LiteLLM for provider-agnostic access. The model string determines
270
  the provider: "claude-*" → Anthropic, "gpt-*" → OpenAI, "gemini/*" → Google, etc.
 
271
  """
 
 
 
272
  import litellm
273
 
274
  # Suppress LiteLLM's verbose logging
@@ -286,10 +435,7 @@ def _call_llm(digest: str, model: str) -> dict:
286
  {"role": "system", "content": _SYSTEM_PROMPT},
287
  {
288
  "role": "user",
289
- "content": (
290
- "Analyze these coding agent sessions and return JSON recommendations:\n\n"
291
- + digest
292
- ),
293
  },
294
  ],
295
  max_tokens=4096,
@@ -298,16 +444,7 @@ def _call_llm(digest: str, model: str) -> dict:
298
 
299
  # Extract text from response
300
  text = response.choices[0].message.content or ""
301
-
302
- # Parse JSON — handle both raw JSON and ```json fenced blocks
303
- text = text.strip()
304
- if text.startswith("```"):
305
- lines = text.split("\n")
306
- lines = [ln for ln in lines[1:] if not ln.strip().startswith("```")]
307
- text = "\n".join(lines)
308
-
309
- result: dict = json.loads(text)
310
- return result
311
 
312
 
313
  # =============================================================================
 
8
 
9
  Supports any LLM provider via LiteLLM: Anthropic, OpenAI, Google, Bedrock,
10
  Ollama, and 100+ others. Auto-detects the best available model from env vars.
11
+ Also supports CLI-based backends (claude, gemini, codex) for subscription
12
+ users without raw API keys.
13
  """
14
 
15
  from __future__ import annotations
 
17
  import json
18
  import logging
19
  import os
20
+ import shutil
21
+ import subprocess
22
 
23
  from .models import (
24
  AnalysisResult,
 
41
 
42
  _MAX_DIGEST_TOKENS = 80_000 # Budget for the digest (leave room for prompt + output)
43
 
44
+ # CLI tools to try when no API key is set (checked in order).
45
+ # Each entry: (binary_name, model_identifier, command_prefix)
46
+ _CLI_BACKENDS: list[tuple[str, str, list[str]]] = [
47
+ ("claude", "claude-cli", ["claude", "-p"]),
48
+ ("gemini", "gemini-cli", ["gemini", "-p"]),
49
+ ("codex", "codex-cli", ["codex", "exec"]),
50
+ ]
51
+
52
+ # Set of valid CLI model identifiers, derived from _CLI_BACKENDS.
53
+ _CLI_MODEL_IDS: set[str] = {model for _, model, _ in _CLI_BACKENDS}
54
+
55
+ _USER_PROMPT_PREFIX = "Analyze these coding agent sessions and return JSON recommendations:\n\n" # Shared by _call_cli_llm and _call_llm
56
+ _MAX_SNIPPET_LEN = 2000 # Max chars of CLI output (stdout/stderr) in error messages
57
+ _CLI_TIMEOUT = 120 # Subprocess timeout for CLI backends, in seconds
58
+
59
 
60
  def _detect_default_model() -> str:
61
+ """Pick the best available model based on API keys, env config, or CLI tools.
62
+
63
+ Priority order:
64
+ 1. API key present → use corresponding LiteLLM model
65
+ 2. HEADROOM_LEARN_CLI env var → use specified CLI backend
66
+ 3. Auto-detect installed CLI tools (claude > gemini > codex)
67
+ 4. Raise RuntimeError with setup instructions
68
+ """
69
+ # 1. API key detection (existing behavior)
70
  for env_var, model in _MODEL_DEFAULTS:
71
  if os.environ.get(env_var):
72
  return model
73
+
74
+ # 2. Explicit CLI selection via environment variable
75
+ cli_override = os.environ.get("HEADROOM_LEARN_CLI")
76
+ if cli_override:
77
+ for cli_name, model, _cmd in _CLI_BACKENDS:
78
+ if cli_name == cli_override:
79
+ logger.info("HEADROOM_LEARN_CLI=%s — using %s CLI backend", cli_override, cli_name)
80
+ return model
81
+ valid = ", ".join(name for name, _, _ in _CLI_BACKENDS)
82
+ raise ValueError(
83
+ f"HEADROOM_LEARN_CLI={cli_override!r} is not a supported CLI. Valid values: {valid}"
84
+ )
85
+
86
+ # 3. Auto-detect installed CLI tools
87
+ for cli_name, model, _cmd in _CLI_BACKENDS:
88
+ if shutil.which(cli_name):
89
+ logger.info("No API key found — auto-detected %s CLI as LLM backend", cli_name)
90
+ return model
91
+
92
  raise RuntimeError(
93
  "No LLM API key found. headroom learn needs one of:\n"
94
  " export ANTHROPIC_API_KEY=sk-ant-... → uses claude-sonnet-4-6\n"
95
  " export OPENAI_API_KEY=sk-... → uses gpt-4o\n"
96
  " export GEMINI_API_KEY=... → uses gemini-2.0-flash\n"
97
+ "Or set HEADROOM_LEARN_CLI to a coding agent CLI (claude, gemini, codex).\n"
98
+ "Or install one of those CLIs for auto-detection.\n"
99
  "Or specify a model directly: headroom learn --model <litellm-model-name>"
100
  )
101
 
 
311
  """
312
 
313
 
314
+ def _strip_fenced_json(raw: str) -> dict:
315
+ """Strip optional markdown fences and parse JSON.
316
+
317
+ Handles both raw JSON and fenced code blocks (e.g. ``​`json ... ``​`).
318
+ Only the first opening fence and last closing fence are removed, preserving
319
+ any triple-backtick content that may appear inside the JSON payload.
320
+
321
+ Args:
322
+ raw: Raw text output from an LLM, possibly wrapped in markdown fences.
323
+
324
+ Returns:
325
+ Parsed JSON as a dictionary.
326
+
327
+ Raises:
328
+ json.JSONDecodeError: If the text is not valid JSON after stripping.
329
+ """
330
+ text = raw.strip()
331
+ if text.startswith("```"):
332
+ lines = text.split("\n")
333
+ # Remove the first line (opening fence, e.g. ```json)
334
+ lines = lines[1:]
335
+ # Remove the last line if it is a closing fence
336
+ if lines and lines[-1].strip().startswith("```"):
337
+ lines = lines[:-1]
338
+ text = "\n".join(lines)
339
+ result: dict = json.loads(text)
340
+ return result
341
+
342
+
343
+ def _call_cli_llm(digest: str, model: str) -> dict:
344
+ """Call a locally installed CLI tool as the LLM backend.
345
+
346
+ Enables keyless usage for subscription-based CLI tools that handle
347
+ their own OAuth authentication. The prompt is passed via stdin to avoid
348
+ OS ``ARG_MAX`` limits and argument-injection risks.
349
+
350
+ CLI invocations:
351
+ claude-cli → echo <prompt> | claude -p
352
+ gemini-cli → echo <prompt> | gemini -p
353
+ codex-cli → echo <prompt> | codex exec
354
+
355
+ Args:
356
+ digest: Token-efficient session digest to analyze.
357
+ model: CLI model identifier (e.g. ``claude-cli``).
358
+
359
+ Returns:
360
+ Parsed JSON recommendations from the CLI tool.
361
+
362
+ Raises:
363
+ ValueError: If *model* is not a known CLI backend.
364
+ RuntimeError: If the CLI exits with a non-zero code or times out.
365
+ """
366
+ cmd: list[str] | None = None
367
+ for _name, model_name, cmd_parts in _CLI_BACKENDS:
368
+ if model_name == model:
369
+ cmd = cmd_parts
370
+ break
371
+ if cmd is None:
372
+ raise ValueError(f"Unknown CLI model: {model}")
373
+
374
+ prompt = _SYSTEM_PROMPT + "\n\n" + _USER_PROMPT_PREFIX + digest
375
+
376
+ try:
377
+ result = subprocess.run(
378
+ cmd,
379
+ input=prompt,
380
+ capture_output=True,
381
+ text=True,
382
+ timeout=_CLI_TIMEOUT,
383
+ )
384
+ except subprocess.TimeoutExpired:
385
+ raise RuntimeError(
386
+ f"`{' '.join(cmd)}` did not respond within {_CLI_TIMEOUT}s. "
387
+ "Check network connectivity or try a different backend with "
388
+ "--model <litellm-model-name>."
389
+ ) from None
390
+
391
+ if result.returncode != 0:
392
+ stderr_snippet = (result.stderr or "")[:_MAX_SNIPPET_LEN]
393
+ raise RuntimeError(
394
+ f"`{' '.join(cmd)}` failed (exit {result.returncode}):\n{stderr_snippet}"
395
+ )
396
+
397
+ # Log stderr warnings even on success (auth refreshes, deprecation notices).
398
+ if result.stderr and result.stderr.strip():
399
+ logger.debug("CLI stderr (exit 0): %s", result.stderr[:_MAX_SNIPPET_LEN])
400
+
401
+ try:
402
+ return _strip_fenced_json(result.stdout)
403
+ except json.JSONDecodeError as exc:
404
+ stdout_snippet = (result.stdout or "")[:_MAX_SNIPPET_LEN]
405
+ raise RuntimeError(
406
+ f"`{' '.join(cmd)}` returned unparseable output. "
407
+ f"First {_MAX_SNIPPET_LEN} chars:\n{stdout_snippet}"
408
+ ) from exc
409
+
410
+
411
  def _call_llm(digest: str, model: str) -> dict:
412
  """Call LLM with the session digest and return parsed JSON.
413
 
414
  Uses LiteLLM for provider-agnostic access. The model string determines
415
  the provider: "claude-*" → Anthropic, "gpt-*" → OpenAI, "gemini/*" → Google, etc.
416
+ For CLI-based models (ending in "-cli"), delegates to ``_call_cli_llm``.
417
  """
418
+ if model in _CLI_MODEL_IDS:
419
+ return _call_cli_llm(digest, model)
420
+
421
  import litellm
422
 
423
  # Suppress LiteLLM's verbose logging
 
435
  {"role": "system", "content": _SYSTEM_PROMPT},
436
  {
437
  "role": "user",
438
+ "content": _USER_PROMPT_PREFIX + digest,
 
 
 
439
  },
440
  ],
441
  max_tokens=4096,
 
444
 
445
  # Extract text from response
446
  text = response.choices[0].message.content or ""
447
+ return _strip_fenced_json(text)
 
 
 
 
 
 
 
 
 
448
 
449
 
450
  # =============================================================================
tests/test_learn/test_analyzer.py CHANGED
@@ -1,13 +1,20 @@
1
  """Tests for session analyzer — digest builder and LLM-based analysis."""
2
 
 
 
3
  from pathlib import Path
4
  from unittest.mock import MagicMock, patch
5
 
 
 
6
  from headroom.learn.analyzer import (
7
  SessionAnalyzer,
8
  _build_digest,
 
 
9
  _detect_default_model,
10
  _parse_llm_response,
 
11
  )
12
  from headroom.learn.models import (
13
  AnalysisResult,
@@ -347,15 +354,230 @@ class TestDetectDefaultModel:
347
  monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
348
  assert _detect_default_model() == "claude-sonnet-4-6"
349
 
350
- def test_no_keys_raises(self, monkeypatch):
351
  monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
352
  monkeypatch.delenv("OPENAI_API_KEY", raising=False)
353
  monkeypatch.delenv("GEMINI_API_KEY", raising=False)
354
- import pytest
355
 
356
  with pytest.raises(RuntimeError, match="No LLM API key found"):
357
  _detect_default_model()
358
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
359
 
360
  # =============================================================================
361
  # Legacy Compatibility
 
1
  """Tests for session analyzer — digest builder and LLM-based analysis."""
2
 
3
+ import json
4
+ import subprocess
5
  from pathlib import Path
6
  from unittest.mock import MagicMock, patch
7
 
8
+ import pytest
9
+
10
  from headroom.learn.analyzer import (
11
  SessionAnalyzer,
12
  _build_digest,
13
+ _call_cli_llm,
14
+ _call_llm,
15
  _detect_default_model,
16
  _parse_llm_response,
17
+ _strip_fenced_json,
18
  )
19
  from headroom.learn.models import (
20
  AnalysisResult,
 
354
  monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
355
  assert _detect_default_model() == "claude-sonnet-4-6"
356
 
357
+ def test_no_keys_no_cli_raises(self, monkeypatch):
358
  monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
359
  monkeypatch.delenv("OPENAI_API_KEY", raising=False)
360
  monkeypatch.delenv("GEMINI_API_KEY", raising=False)
361
+ monkeypatch.setattr("headroom.learn.analyzer.shutil.which", lambda _name: None)
362
 
363
  with pytest.raises(RuntimeError, match="No LLM API key found"):
364
  _detect_default_model()
365
 
366
+ def test_cli_fallback_claude(self, monkeypatch):
367
+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
368
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
369
+ monkeypatch.delenv("GEMINI_API_KEY", raising=False)
370
+ monkeypatch.setattr(
371
+ "headroom.learn.analyzer.shutil.which",
372
+ lambda name: f"/usr/bin/{name}" if name == "claude" else None,
373
+ )
374
+ assert _detect_default_model() == "claude-cli"
375
+
376
+ def test_cli_fallback_gemini(self, monkeypatch):
377
+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
378
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
379
+ monkeypatch.delenv("GEMINI_API_KEY", raising=False)
380
+ monkeypatch.setattr(
381
+ "headroom.learn.analyzer.shutil.which",
382
+ lambda name: f"/usr/bin/{name}" if name == "gemini" else None,
383
+ )
384
+ assert _detect_default_model() == "gemini-cli"
385
+
386
+ def test_cli_fallback_codex(self, monkeypatch):
387
+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
388
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
389
+ monkeypatch.delenv("GEMINI_API_KEY", raising=False)
390
+ monkeypatch.setattr(
391
+ "headroom.learn.analyzer.shutil.which",
392
+ lambda name: f"/usr/bin/{name}" if name == "codex" else None,
393
+ )
394
+ assert _detect_default_model() == "codex-cli"
395
+
396
+ def test_api_key_preferred_over_cli(self, monkeypatch):
397
+ monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test")
398
+ monkeypatch.setattr(
399
+ "headroom.learn.analyzer.shutil.which",
400
+ lambda name: f"/usr/bin/{name}" if name == "claude" else None,
401
+ )
402
+ assert _detect_default_model() == "claude-sonnet-4-6"
403
+
404
+ def test_env_var_selects_gemini(self, monkeypatch):
405
+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
406
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
407
+ monkeypatch.delenv("GEMINI_API_KEY", raising=False)
408
+ monkeypatch.setenv("HEADROOM_LEARN_CLI", "gemini")
409
+ assert _detect_default_model() == "gemini-cli"
410
+
411
+ def test_env_var_selects_codex(self, monkeypatch):
412
+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
413
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
414
+ monkeypatch.delenv("GEMINI_API_KEY", raising=False)
415
+ monkeypatch.setenv("HEADROOM_LEARN_CLI", "codex")
416
+ assert _detect_default_model() == "codex-cli"
417
+
418
+ def test_env_var_invalid_raises(self, monkeypatch):
419
+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
420
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
421
+ monkeypatch.delenv("GEMINI_API_KEY", raising=False)
422
+ monkeypatch.setenv("HEADROOM_LEARN_CLI", "unknown-tool")
423
+ with pytest.raises(ValueError, match="not a supported CLI"):
424
+ _detect_default_model()
425
+
426
+ def test_api_key_preferred_over_env_var(self, monkeypatch):
427
+ monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test")
428
+ monkeypatch.setenv("HEADROOM_LEARN_CLI", "gemini")
429
+ assert _detect_default_model() == "claude-sonnet-4-6"
430
+
431
+ def test_env_var_preferred_over_auto_detect(self, monkeypatch):
432
+ monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
433
+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)
434
+ monkeypatch.delenv("GEMINI_API_KEY", raising=False)
435
+ monkeypatch.setenv("HEADROOM_LEARN_CLI", "codex")
436
+ monkeypatch.setattr(
437
+ "headroom.learn.analyzer.shutil.which",
438
+ lambda name: f"/usr/bin/{name}" if name == "claude" else None,
439
+ )
440
+ # codex selected via env var, even though claude is in PATH
441
+ assert _detect_default_model() == "codex-cli"
442
+
443
+
444
+ # =============================================================================
445
+ # CLI LLM Backend
446
+ # =============================================================================
447
+
448
+
449
+ class TestStripFencedJson:
450
+ def test_raw_json(self):
451
+ result = _strip_fenced_json('{"key": "value"}')
452
+ assert result == {"key": "value"}
453
+
454
+ def test_fenced_json(self):
455
+ raw = '```json\n{"key": "value"}\n```'
456
+ result = _strip_fenced_json(raw)
457
+ assert result == {"key": "value"}
458
+
459
+ def test_fenced_no_language_tag(self):
460
+ raw = '```\n{"key": "value"}\n```'
461
+ result = _strip_fenced_json(raw)
462
+ assert result == {"key": "value"}
463
+
464
+ def test_whitespace_padding(self):
465
+ raw = ' \n```json\n{"key": "value"}\n```\n '
466
+ result = _strip_fenced_json(raw)
467
+ assert result == {"key": "value"}
468
+
469
+ def test_invalid_json_raises(self):
470
+ with pytest.raises(json.JSONDecodeError):
471
+ _strip_fenced_json("not json at all")
472
+
473
+
474
+ class TestCallCliLlm:
475
+ @patch("headroom.learn.analyzer.subprocess.run")
476
+ def test_claude_cli_success(self, mock_run: MagicMock):
477
+ mock_run.return_value = MagicMock(
478
+ returncode=0,
479
+ stdout='{"context_file_rules": [], "memory_file_rules": []}',
480
+ stderr="",
481
+ )
482
+ result = _call_cli_llm("test digest", "claude-cli")
483
+ assert result == {"context_file_rules": [], "memory_file_rules": []}
484
+ mock_run.assert_called_once()
485
+ cmd = mock_run.call_args[0][0]
486
+ assert cmd == ["claude", "-p"]
487
+ # Prompt passed via stdin, not as an argument
488
+ assert mock_run.call_args.kwargs.get("input") is not None
489
+
490
+ @patch("headroom.learn.analyzer.subprocess.run")
491
+ def test_codex_cli_uses_exec(self, mock_run: MagicMock):
492
+ mock_run.return_value = MagicMock(
493
+ returncode=0,
494
+ stdout='{"context_file_rules": [], "memory_file_rules": []}',
495
+ stderr="",
496
+ )
497
+ result = _call_cli_llm("test digest", "codex-cli")
498
+ assert result == {"context_file_rules": [], "memory_file_rules": []}
499
+ cmd = mock_run.call_args[0][0]
500
+ assert cmd == ["codex", "exec"]
501
+
502
+ @patch("headroom.learn.analyzer.subprocess.run")
503
+ def test_gemini_cli_uses_p_flag(self, mock_run: MagicMock):
504
+ mock_run.return_value = MagicMock(
505
+ returncode=0,
506
+ stdout='{"context_file_rules": [], "memory_file_rules": []}',
507
+ stderr="",
508
+ )
509
+ _call_cli_llm("test digest", "gemini-cli")
510
+ cmd = mock_run.call_args[0][0]
511
+ assert cmd == ["gemini", "-p"]
512
+
513
+ @patch("headroom.learn.analyzer.subprocess.run")
514
+ def test_cli_nonzero_exit_raises(self, mock_run: MagicMock):
515
+ mock_run.return_value = MagicMock(
516
+ returncode=1,
517
+ stdout="",
518
+ stderr="Error: auth required",
519
+ )
520
+ with pytest.raises(RuntimeError, match="failed.*exit 1"):
521
+ _call_cli_llm("test digest", "claude-cli")
522
+
523
+ @patch("headroom.learn.analyzer.subprocess.run")
524
+ def test_cli_stderr_truncated_in_error(self, mock_run: MagicMock):
525
+ long_stderr = "x" * 5000
526
+ mock_run.return_value = MagicMock(
527
+ returncode=1,
528
+ stdout="",
529
+ stderr=long_stderr,
530
+ )
531
+ with pytest.raises(RuntimeError) as exc_info:
532
+ _call_cli_llm("test digest", "claude-cli")
533
+ # Full 5000-char stderr should not appear in the error message
534
+ assert long_stderr not in str(exc_info.value)
535
+
536
+ def test_unknown_cli_model_raises(self):
537
+ with pytest.raises(ValueError, match="Unknown CLI model"):
538
+ _call_cli_llm("test digest", "unknown-cli")
539
+
540
+ @patch("headroom.learn.analyzer.subprocess.run")
541
+ def test_fenced_output_parsed(self, mock_run: MagicMock):
542
+ mock_run.return_value = MagicMock(
543
+ returncode=0,
544
+ stdout='```json\n{"context_file_rules": [], "memory_file_rules": []}\n```',
545
+ stderr="",
546
+ )
547
+ result = _call_cli_llm("test digest", "claude-cli")
548
+ assert result == {"context_file_rules": [], "memory_file_rules": []}
549
+
550
+ @patch("headroom.learn.analyzer.subprocess.run")
551
+ def test_timeout_raises_runtime_error(self, mock_run: MagicMock):
552
+ mock_run.side_effect = subprocess.TimeoutExpired(cmd=["claude", "-p"], timeout=120)
553
+ with pytest.raises(RuntimeError, match="did not respond within"):
554
+ _call_cli_llm("test digest", "claude-cli")
555
+
556
+ @patch("headroom.learn.analyzer.subprocess.run")
557
+ def test_unparseable_output_raises_with_context(self, mock_run: MagicMock):
558
+ mock_run.return_value = MagicMock(
559
+ returncode=0,
560
+ stdout="This is not JSON at all",
561
+ stderr="",
562
+ )
563
+ with pytest.raises(RuntimeError, match="unparseable output"):
564
+ _call_cli_llm("test digest", "claude-cli")
565
+
566
+
567
+ class TestCallLlmRouting:
568
+ @patch("headroom.learn.analyzer._call_cli_llm")
569
+ def test_routes_cli_model_to_cli_backend(self, mock_cli: MagicMock):
570
+ mock_cli.return_value = {"context_file_rules": [], "memory_file_rules": []}
571
+ result = _call_llm("test digest", "claude-cli")
572
+ mock_cli.assert_called_once_with("test digest", "claude-cli")
573
+ assert result == {"context_file_rules": [], "memory_file_rules": []}
574
+
575
+ @patch("headroom.learn.analyzer._call_cli_llm")
576
+ def test_routes_codex_cli(self, mock_cli: MagicMock):
577
+ mock_cli.return_value = {}
578
+ _call_llm("digest", "codex-cli")
579
+ mock_cli.assert_called_once_with("digest", "codex-cli")
580
+
581
 
582
  # =============================================================================
583
  # Legacy Compatibility