chopratejas commited on
Commit
ff29454
·
1 Parent(s): 0fb4bed

README: add back proof points — needle-in-haystack demo, benchmarks, architecture detail

Browse files
Files changed (1) hide show
  1. README.md +41 -9
README.md CHANGED
@@ -89,6 +89,37 @@ You don't need to replace it. Drop Headroom into your existing stack:
89
 
90
  ---
91
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
  ## How It Works
93
 
94
  ```
@@ -99,22 +130,21 @@ Your App → Headroom → LLM Provider
99
  → SmartCrusher (JSON) | CodeCompressor (code) | LLMLingua (text)
100
  IntelligentContext: score-based token fitting
101
  CCR: stores originals for retrieval if LLM needs more
 
102
  ```
103
 
104
- Headroom never throws data away. It compresses aggressively and retrieves precisely.
105
 
106
- ---
107
-
108
- ## Verified Performance
109
 
110
- | Scenario | Tokens Before | Tokens After | Savings |
111
- |----------|--------------|-------------|---------|
112
  | Code search (100 results) | 17,765 | 1,408 | **92%** |
113
  | SRE incident debugging | 65,694 | 5,118 | **92%** |
114
  | Codebase exploration | 78,502 | 41,254 | **47%** |
115
  | GitHub issue triage | 54,174 | 14,761 | **73%** |
116
 
117
- **Overhead**: 1-5ms. **Accuracy**: [benchmarked](docs/benchmarks.md) across 12+ datasets.
118
 
119
  ---
120
 
@@ -138,10 +168,12 @@ Headroom never throws data away. It compresses aggressively and retrieves precis
138
  | Feature | What it does |
139
  |---------|-------------|
140
  | **Content Router** | Auto-detects content type, routes to optimal compressor |
141
- | **SmartCrusher** | Statistically compresses JSON arrays (tool outputs, API responses) |
142
- | **CodeCompressor** | AST-aware code compression (Python, JS, Go, Rust, Java) |
143
  | **LLMLingua-2** | ML-based 20x text compression |
144
  | **CCR** | Reversible compression — LLM retrieves originals when needed |
 
 
145
  | **CacheAligner** | Stabilizes prefixes for provider KV cache hits |
146
  | **IntelligentContext** | Score-based context management with learned importance |
147
  | **Image Compression** | 40-90% token reduction via trained ML router |
 
89
 
90
  ---
91
 
92
+ ## Does It Actually Work?
93
+
94
+ **100 production log entries. One critical error buried at position 67.**
95
+
96
+ | | Baseline | Headroom |
97
+ |--|----------|----------|
98
+ | Input tokens | 10,144 | 1,260 |
99
+ | Correct answers | **4/4** | **4/4** |
100
+
101
+ Both responses: *"payment-gateway, error PG-5523, fix: Increase max_connections to 500, 1,847 transactions affected."*
102
+
103
+ **87.6% fewer tokens. Same answer.** Run it: `python examples/needle_in_haystack_test.py`
104
+
105
+ <details>
106
+ <summary><b>What Headroom kept</b></summary>
107
+
108
+ From 100 log entries, SmartCrusher kept 6: first 3 (boundary), the FATAL error at position 67 (anomaly detection), and last 2 (recency). The error was automatically preserved — not by keyword matching, but by statistical analysis of field variance.
109
+ </details>
110
+
111
+ ### Accuracy Benchmarks
112
+
113
+ | Benchmark | Metric | Result | Compression |
114
+ |-----------|--------|--------|------------|
115
+ | [Scrapinghub Extraction](https://huggingface.co/datasets/allenai/scrapinghub-article-extraction-benchmark) | Recall | **98.2%** | 94.9% |
116
+ | Multi-Tool Agent (4 tools) | Accuracy | **100%** | 76.3% |
117
+ | SmartCrusher (JSON) | Accuracy | **100%** | 87.6% |
118
+
119
+ Full methodology: [Benchmarks](docs/benchmarks.md) | Run yourself: `python -m headroom.evals quick`
120
+
121
+ ---
122
+
123
  ## How It Works
124
 
125
  ```
 
130
  → SmartCrusher (JSON) | CodeCompressor (code) | LLMLingua (text)
131
  IntelligentContext: score-based token fitting
132
  CCR: stores originals for retrieval if LLM needs more
133
+ Query Echo: re-injects user question for fresh attention after compression
134
  ```
135
 
136
+ Headroom never throws data away. It compresses aggressively and retrieves precisely. When it compresses 500 items to 20, it tells the LLM *what was omitted* ("87 passed, 2 failed, 1 error") so the LLM knows when to ask for more.
137
 
138
+ ### Verified on Real Workloads
 
 
139
 
140
+ | Scenario | Before | After | Savings |
141
+ |----------|--------|-------|---------|
142
  | Code search (100 results) | 17,765 | 1,408 | **92%** |
143
  | SRE incident debugging | 65,694 | 5,118 | **92%** |
144
  | Codebase exploration | 78,502 | 41,254 | **47%** |
145
  | GitHub issue triage | 54,174 | 14,761 | **73%** |
146
 
147
+ **Overhead**: 1-5ms compression latency.
148
 
149
  ---
150
 
 
168
  | Feature | What it does |
169
  |---------|-------------|
170
  | **Content Router** | Auto-detects content type, routes to optimal compressor |
171
+ | **SmartCrusher** | Statistically compresses JSON arrays preserves errors, anomalies, boundaries |
172
+ | **CodeCompressor** | AST-aware compression for Python, JS, Go, Rust, Java, C++ |
173
  | **LLMLingua-2** | ML-based 20x text compression |
174
  | **CCR** | Reversible compression — LLM retrieves originals when needed |
175
+ | **Compression Summaries** | Tells the LLM what was omitted ("3 errors, 12 failures") |
176
+ | **Query Echo** | Re-injects user question after compressed data for better attention |
177
  | **CacheAligner** | Stabilizes prefixes for provider KV cache hits |
178
  | **IntelligentContext** | Score-based context management with learned importance |
179
  | **Image Compression** | 40-90% token reduction via trained ML router |