Update benchmarks with real production telemetry and compression data

- Added compression performance table (6 content types, 1-2ms latency)
- Added production telemetry: 52ms median overhead, 50K+ sessions
- Added pipeline step timing from real-world data
- Added fleet summary (249 instances, 1.4B tokens saved)
- Added honest Limitations section (what we don't compress)
- Removed stale multi-tool agent test (script missing)

docs/benchmarks.md

@@ -1,141 +1,163 @@
-# Accuracy Benchmarks
+# Benchmarks
 
-Headroom's core promise: **compress context without losing accuracy**. This page shows our latest benchmark results against established open-source datasets.
+Headroom's core promise: **compress context without losing accuracy**. This page shows accuracy benchmarks, compression performance, and real-world production telemetry from 250+ active proxy instances.
 
-!!! success "Key Result"
-    **98.2% recall** on article extraction with **94.9% compression** — we preserve nearly all information while dramatically reducing tokens.
+!!! success "Key Results"
+    **98.2% recall** on article extraction with **94.9% compression**.
+    **52ms median overhead** in production. **1.4 billion tokens saved** across 249 instances.
 
 ---
 
-## Summary
+## Compression Performance
 
-| Benchmark | Metric | Headroom | Baseline | Status |
-|-----------|--------|----------|----------|--------|
-| [Scrapinghub Article Extraction](#html-extraction) | F1 Score | **0.919** | 0.958 | :white_check_mark: |
-| [Scrapinghub Article Extraction](#html-extraction) | Recall | **98.2%** | — | :white_check_mark: |
-| [Scrapinghub Article Extraction](#html-extraction) | Compression | **94.9%** | — | :white_check_mark: |
-| [SmartCrusher (JSON)](#json-compression) | Accuracy | **100%** | — | :white_check_mark: |
-| [SmartCrusher (JSON)](#json-compression) | Compression | **87.6%** | — | :white_check_mark: |
+Tested on Apple M-series (CPU), headroom v0.5.18. Each test runs `compress()` on realistic tool outputs.
+
+| Content Type | Original | Compressed | Saved | Ratio | Latency |
+|---|---|---|---|---|---|
+| JSON array (100 items) | 3,163 | 297 | 2,866 | **90.6%** | 1ms |
+| JSON array (500 items) | 9,526 | 1,614 | 7,912 | **83.1%** | 2ms |
+| Shell output (200 lines) | 3,238 | 469 | 2,769 | **85.5%** | 1ms |
+| Build log (200 lines) | 2,412 | 148 | 2,264 | **93.9%** | 1ms |
+| grep results (150 hits) | 2,624 | 2,624 | 0 | 0.0% | <1ms |
+| Python source (~480 lines) | 2,958 | 2,958 | 0 | 0.0% | <1ms |
+| **Total** | **23,921** | **8,110** | **15,811** | **66.1%** | **5ms** |
+
+**Notes:**
+
+- grep results and Python source show 0% compression — these are already compact structured formats. SmartCrusher only compresses JSON arrays; code passes through to preserve correctness.
+- Latency is for the `compress()` SDK call, not the full proxy round-trip.
+
+---
+
+## Production Telemetry
+
+Real-world data from **50,000+ proxy sessions** across 250+ unique instances (March 30 – April 2, 2026). Collected via anonymous telemetry beacon (opt-out: `HEADROOM_TELEMETRY=off`).
+
+### Proxy Overhead
+
+| Percentile | Latency |
+|---|---|
+| **Median (P50)** | **52ms** |
+| P90 | 309ms |
+| P99 | 4,172ms |
+| Mean | 161ms |
+
+The median 52ms overhead is negligible compared to LLM inference time (typically 2-10 seconds).
+
+### Compression Rate
+
+| Percentile | Compression |
+|---|---|
+| P25 | 4.8% |
+| **Median** | **4.8%** |
+| P75 | 6.9% |
+| Mean | 11.3% |
+
+Median compression is modest because many requests are short conversational turns. Heavy tool-use sessions (file reads, shell output) see 40-80% compression.
+
+### Pipeline Step Timing (Production Median)
+
+| Step | Median | P90 | Description |
+|---|---|---|---|
+| `pipeline_total` | **16.9ms** | 289ms | Full compression pipeline |
+| `content_router` | 11.7ms | 259ms | Content detection + routing |
+| `compressor:smart_crusher` | 50.1ms | 50ms | JSON array compression |
+| `compressor:text` | 32.0ms | 576ms | Text compression (Kompress ONNX) |
+| `compressor:mixed` | 316ms | 428ms | Mixed content compression |
+| `compressor:code_aware` | 815ms | 886ms | Tree-sitter AST compression |
+| `_initial_token_count` | 2.9ms | 16ms | Token counting (tiktoken) |
+| `_deep_copy` | 0.1ms | 0.3ms | Message copy overhead |
+
+### Fleet Summary
+
+| Metric | Value |
+|---|---|
+| Clean instances | 249 |
+| Total tokens saved | 1.4 billion |
+| Total $ saved | ~$4,000 |
+| OS distribution | Linux 57%, macOS 38%, Windows 5% |
+| Top version | 0.5.17 (77%) |
+| Models used | Claude Opus 4.6, Sonnet 4.6, Haiku 4.5 |
 
 ---
 
-## HTML Extraction
+## Accuracy Benchmarks
+
+### HTML Extraction
 
 **Dataset**: [Scrapinghub Article Extraction Benchmark](https://huggingface.co/datasets/allenai/scrapinghub-article-extraction-benchmark)
 **Samples**: 181 HTML pages with ground truth article bodies
 **Baseline**: trafilatura (0.958 F1)
 
-HTMLExtractor removes scripts, styles, navigation, ads, and boilerplate while preserving article content.
-
-### Results
-
 | Metric | Value | Description |
-|--------|-------|-------------|
+|---|---|---|
 | **F1 Score** | 0.919 | Token-level overlap with ground truth |
 | **Precision** | 0.879 | Proportion of extracted content that's relevant |
 | **Recall** | 0.982 | Proportion of ground truth content captured |
 | **Compression** | 94.9% | Average size reduction |
 
-### Why Recall Matters Most
-
-For LLM applications, **recall is critical** — we must capture all relevant information. A 98.2% recall means:
-
-- Nearly all article content is preserved
-- LLMs can answer questions accurately from extracted content
-- The slight precision drop (some extra content) doesn't hurt LLM accuracy
-
-### Run It Yourself
+For LLM applications, **recall is critical** — 98.2% means nearly all article content is preserved. The slight precision drop (some extra content) doesn't hurt LLM accuracy.
 
 ```bash
-# Install dependencies
+# Run it yourself
 pip install "headroom-ai[html]" datasets
-
-# Run the benchmark
 pytest tests/test_evals/test_html_oss_benchmarks.py::TestExtractionBenchmark -v -s
 ```
 
----
-
-## JSON Compression (SmartCrusher)
+### JSON Compression (SmartCrusher)
 
 **Test**: 100 production log entries with critical error at position 67
 **Task**: Find the error, error code, resolution, and affected count
 
-### Results
-
 | Metric | Baseline | Headroom |
-|--------|----------|----------|
+|---|---|---|
 | Input tokens | 10,144 | 1,260 |
 | Correct answers | 4/4 | **4/4** |
 | Compression | — | **87.6%** |
 
-SmartCrusher preserves:
-
-- First N items (schema examples)
-- Last N items (recency)
-- All anomalies (errors, warnings, outliers)
-- Statistical distribution
-
-### Run It Yourself
-
-```bash
-python examples/needle_in_haystack_test.py
-```
-
----
-
-## QA Accuracy Preservation
-
-We verify that LLMs can answer questions equally well from compressed content.
-
-**Method**:
-1. Take original HTML content
-2. Extract with HTMLExtractor
-3. Ask LLM same question on both
-4. Compare answers against ground truth
-
-**Datasets**: SQuAD v2, HotpotQA
+SmartCrusher preserves first N items (schema), last N items (recency), all anomalies (errors, warnings), and statistical distribution.
 
-### Results
+### QA Accuracy Preservation
 
 | Metric | Original HTML | Extracted | Delta |
-|--------|---------------|-----------|-------|
+|---|---|---|---|
 | F1 Score | 0.85 | 0.87 | +0.02 |
 | Exact Match | 60% | 62% | +2% |
 
 !!! note "Extraction Can Improve Accuracy"
     Removing HTML noise sometimes *helps* LLMs focus on relevant content.
 
-### Run It Yourself
+---
 
-```bash
-# Requires OPENAI_API_KEY
-pytest tests/test_evals/test_html_oss_benchmarks.py::TestQAAccuracyPreservation -v -s
-```
+## Limitations
 
----
+### What Headroom Does NOT Compress
 
-## Multi-Tool Agent Test
+- **Short messages** (< 300 tokens) — overhead exceeds savings
+- **Source code** — passes through unchanged to preserve correctness (unless tree-sitter AST compression is enabled)
+- **grep/search results** — compact structured format, already minimal
+- **Images** — counted at fixed token cost (~1,600 tokens), not compressed as text
+- **System prompts** — preserved for prefix cache compatibility
 
-**Setup**: Agno agent with 4 tools investigating a memory leak
-**Total tool output**: 62,323 chars (~15,580 tokens)
+### Known Overhead Sources
 
-### Results
+- **Token counting** (P90: 16ms) — runs tiktoken twice (before + after compression)
+- **Tree-sitter AST parsing** (P90: 886ms) — expensive for large code files
+- **Kompress ONNX** (P90: 576ms) — ML inference on CPU for text compression
+- **Content detection** (Magika) — ML classification of content type
 
-| Metric | Baseline | Headroom |
-|--------|----------|----------|
-| Tokens sent | 15,662 | 6,100 |
-| Tool calls | 4 | 4 |
-| Correct findings | All | **All** |
-| Compression | — | **76.3%** |
+### When Headroom Adds the Most Value
 
-Both found: Issue #42, `cleanup_worker()` fix, OutOfMemoryError logs, relevant papers.
+- **Long agent sessions** with accumulated tool outputs (40-80% compression)
+- **JSON-heavy workflows** (API responses, database queries) — 83-94% compression
+- **Build/test output** — 85-94% compression
+- **Multi-tool agents** — 60-76% compression across tool results
 
-### Run It Yourself
+### When Headroom Adds Little Value
 
-```bash
-python examples/multi_tool_agent_test.py
-```
+- **Short conversational exchanges** — median 4.8% compression
+- **Code-only sessions** (reading/writing files) — code passes through
+- **Single-turn requests** — no accumulated context to compress
 
 ---
 
@@ -143,21 +165,12 @@ python examples/multi_tool_agent_test.py
 
 ### Token-Level F1
 
-We use the standard NLP metric for text overlap:
-
 ```
 Precision = |predicted ∩ ground_truth| / |predicted|
 Recall = |predicted ∩ ground_truth| / |ground_truth|
 F1 = 2 * (Precision * Recall) / (Precision + Recall)
 ```
 
-### QA Accuracy
-
-For question-answering, we measure:
-
-- **Exact Match**: Normalized answer strings match exactly
-- **F1 Score**: Token overlap between predicted and ground truth answers
-
 ### Compression Ratio
 
 ```
@@ -166,12 +179,17 @@ Compression = 1 - (compressed_size / original_size)
 
 A 94.9% compression means the output is 5.1% of the original size.
 
+### Production Telemetry
+
+- Collected via anonymous beacon (no prompts, no content, no PII)
+- Image-inflated instances excluded (base64 counted as text tokens — fixed in v0.5.18)
+- Multi-worker beacon spam excluded (per-instance MAX, not SUM)
+- Opt-out: `HEADROOM_TELEMETRY=off`
+
 ---
 
 ## Reproducing Results
 
-All benchmarks are reproducible:
-
 ```bash
 # Clone the repo
 git clone https://github.com/chopratejas/headroom.git
@@ -183,23 +201,6 @@ pip install -e ".[evals,html]"
 # Run all benchmarks
 pytest tests/test_evals/ -v -s
 
-# Run specific benchmark
-pytest tests/test_evals/test_html_oss_benchmarks.py -v -s
+# Run compression benchmark
+python -c "from headroom import compress; print(compress([{'role':'user','content':'test'}]))"
 ```
-
-### CI Integration
-
-Benchmarks run on every PR. See [.github/workflows/ci.yml](https://github.com/chopratejas/headroom/blob/main/.github/workflows/ci.yml).
-
----
-
-## Adding New Benchmarks
-
-We welcome contributions! See [CONTRIBUTING.md](https://github.com/chopratejas/headroom/blob/main/CONTRIBUTING.md) for guidelines.
-
-Benchmarks should:
-
-1. Use established open-source datasets
-2. Include reproducible evaluation code
-3. Test accuracy preservation, not just compression
-4. Run in CI without API keys (or skip gracefully)