Spaces:
Build error
Build error
Commit ·
8455c4a
1
Parent(s): b6662f2
Add image token compression with trained ML router
Browse filesIntroduces automatic image compression for LLM requests, reducing token
usage by 40-90% while maintaining answer accuracy.
Key features:
- Trained MiniLM classifier (93.7% accuracy) hosted on HuggingFace
- SigLIP-based image analysis for content-aware routing
- Provider-specific compression:
- OpenAI: detail="low" parameter
- Anthropic: PIL resize to 512px
- Google: PIL resize to 768px (tile-optimized)
- Four compression techniques: full_low, preserve, crop, transcode
- Integration in both Headroom proxy and SDK (ContentRouter)
New files:
- headroom/image/ module with ImageCompressor API
- docs/image-compression.md user documentation
- tests/test_image_compressor.py (51 tests)
Model: chopratejas/technique-router on HuggingFace (~128MB)
- README.md +3 -1
- docs/ARCHITECTURE.md +96 -0
- docs/README.md +1 -0
- docs/image-compression.md +318 -0
- headroom/image/__init__.py +47 -0
- headroom/image/compressor.py +450 -0
- headroom/image/trained_router.py +328 -0
- headroom/proxy/server.py +43 -0
- headroom/transforms/content_router.py +74 -0
- pyproject.toml +5 -0
- tests/test_image_compressor.py +927 -0
- uv.lock +0 -0
README.md
CHANGED
|
@@ -186,8 +186,9 @@ For deep technical details, see [Architecture Documentation](docs/ARCHITECTURE.m
|
|
| 186 |
|
| 187 |
- **Zero code changes** - works as a transparent proxy
|
| 188 |
- **47-92% savings** - depends on your workload (tool-heavy = more savings)
|
|
|
|
| 189 |
- **Reversible compression** - LLM retrieves original data via CCR
|
| 190 |
-
- **Content-aware** - code, logs, JSON each handled optimally
|
| 191 |
- **Provider caching** - automatic prefix optimization for cache hits
|
| 192 |
- **Framework native** - LangChain, Agno, MCP, agents supported
|
| 193 |
|
|
@@ -272,6 +273,7 @@ See the full [Agno Integration Guide](docs/agno.md) for hooks, multi-provider su
|
|
| 272 |
|
| 273 |
| Feature | Description | Docs |
|
| 274 |
|---------|-------------|------|
|
|
|
|
| 275 |
| **Memory** | Persistent memory across conversations (zero-latency inline extraction) | [Memory](docs/memory.md) |
|
| 276 |
| **Universal Compression** | ML-based content detection + structure-preserving compression | [Compression](docs/compression.md) |
|
| 277 |
| **SmartCrusher** | Compresses JSON tool outputs statistically | [Transforms](docs/transforms.md) |
|
|
|
|
| 186 |
|
| 187 |
- **Zero code changes** - works as a transparent proxy
|
| 188 |
- **47-92% savings** - depends on your workload (tool-heavy = more savings)
|
| 189 |
+
- **Image compression** - 40-90% reduction via trained ML router (OpenAI, Anthropic, Google)
|
| 190 |
- **Reversible compression** - LLM retrieves original data via CCR
|
| 191 |
+
- **Content-aware** - code, logs, JSON, images each handled optimally
|
| 192 |
- **Provider caching** - automatic prefix optimization for cache hits
|
| 193 |
- **Framework native** - LangChain, Agno, MCP, agents supported
|
| 194 |
|
|
|
|
| 273 |
|
| 274 |
| Feature | Description | Docs |
|
| 275 |
|---------|-------------|------|
|
| 276 |
+
| **Image Compression** | 40-90% token reduction for images via trained ML router | [Image Compression](docs/image-compression.md) |
|
| 277 |
| **Memory** | Persistent memory across conversations (zero-latency inline extraction) | [Memory](docs/memory.md) |
|
| 278 |
| **Universal Compression** | ML-based content detection + structure-preserving compression | [Compression](docs/compression.md) |
|
| 279 |
| **SmartCrusher** | Compresses JSON tool outputs statistically | [Transforms](docs/transforms.md) |
|
docs/ARCHITECTURE.md
CHANGED
|
@@ -931,6 +931,102 @@ class ContextTrackerConfig:
|
|
| 931 |
|
| 932 |
---
|
| 933 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 934 |
## File Structure Explained
|
| 935 |
|
| 936 |
```
|
|
|
|
| 931 |
|
| 932 |
---
|
| 933 |
|
| 934 |
+
## Image Compression Architecture
|
| 935 |
+
|
| 936 |
+
Vision models charge by the token, and images are expensive (765-2900 tokens for a typical image). Headroom's image compression uses a **trained ML router** to automatically select the optimal compression technique.
|
| 937 |
+
|
| 938 |
+
### The Key Insight
|
| 939 |
+
|
| 940 |
+
Not all image queries need full resolution:
|
| 941 |
+
- "What is this?" → Low detail is fine (87% savings)
|
| 942 |
+
- "Count the whiskers" → Need full detail (0% savings)
|
| 943 |
+
- "Read the sign" → Could convert to text (99% savings)
|
| 944 |
+
|
| 945 |
+
### How It Works
|
| 946 |
+
|
| 947 |
+
```
|
| 948 |
+
User: [image] + "What animal is this?"
|
| 949 |
+
↓
|
| 950 |
+
┌─────────────────────────────────┐
|
| 951 |
+
│ 1. Query Analysis │
|
| 952 |
+
│ TrainedRouter (MiniLM) │
|
| 953 |
+
│ Classifies → full_low │
|
| 954 |
+
└─────────────────────────────────┘
|
| 955 |
+
↓
|
| 956 |
+
┌─────────────────────────────────┐
|
| 957 |
+
│ 2. Image Analysis (Optional) │
|
| 958 |
+
│ SigLIP checks: │
|
| 959 |
+
│ - Has text? Is complex? │
|
| 960 |
+
│ - Fine details needed? │
|
| 961 |
+
└─────────────────────────────────┘
|
| 962 |
+
↓
|
| 963 |
+
┌─────────────────────────────────┐
|
| 964 |
+
│ 3. Apply Compression │
|
| 965 |
+
│ OpenAI: detail="low" │
|
| 966 |
+
│ Anthropic: Resize to 512px │
|
| 967 |
+
│ Google: Resize to 768px │
|
| 968 |
+
└─────────────────────────────────┘
|
| 969 |
+
↓
|
| 970 |
+
Compressed request → LLM → Response
|
| 971 |
+
```
|
| 972 |
+
|
| 973 |
+
### The Trained Router
|
| 974 |
+
|
| 975 |
+
A fine-tuned MiniLM classifier hosted on HuggingFace:
|
| 976 |
+
|
| 977 |
+
- **Model**: `chopratejas/technique-router`
|
| 978 |
+
- **Size**: ~128MB (downloaded once, cached)
|
| 979 |
+
- **Accuracy**: 93.7% on 1,157 training examples
|
| 980 |
+
- **Latency**: ~10ms CPU, ~2ms GPU
|
| 981 |
+
|
| 982 |
+
The router learns from examples like:
|
| 983 |
+
| Query | Technique |
|
| 984 |
+
|-------|-----------|
|
| 985 |
+
| "What is this?" | `full_low` |
|
| 986 |
+
| "Count the items" | `preserve` |
|
| 987 |
+
| "Read the text" | `transcode` |
|
| 988 |
+
| "What's in the corner?" | `crop` |
|
| 989 |
+
|
| 990 |
+
### Provider-Specific Compression
|
| 991 |
+
|
| 992 |
+
Each provider handles images differently:
|
| 993 |
+
|
| 994 |
+
| Provider | Method | Savings |
|
| 995 |
+
|----------|--------|---------|
|
| 996 |
+
| **OpenAI** | `detail="low"` parameter | ~87% |
|
| 997 |
+
| **Anthropic** | PIL resize to 512px | ~75% |
|
| 998 |
+
| **Google** | PIL resize to 768px (tile-optimized) | ~75% |
|
| 999 |
+
|
| 1000 |
+
### Integration Points
|
| 1001 |
+
|
| 1002 |
+
Image compression runs in the proxy **before** text compression:
|
| 1003 |
+
|
| 1004 |
+
```
|
| 1005 |
+
Request arrives
|
| 1006 |
+
↓
|
| 1007 |
+
[Image Compression] ← NEW
|
| 1008 |
+
↓
|
| 1009 |
+
[Transform Pipeline: Cache Aligner → Smart Crusher → ...]
|
| 1010 |
+
↓
|
| 1011 |
+
Forward to LLM
|
| 1012 |
+
```
|
| 1013 |
+
|
| 1014 |
+
This ensures images are compressed first, then text compression (CCR, SmartCrusher) handles the rest.
|
| 1015 |
+
|
| 1016 |
+
### Code Location
|
| 1017 |
+
|
| 1018 |
+
```
|
| 1019 |
+
headroom/
|
| 1020 |
+
├── image/
|
| 1021 |
+
│ ├── __init__.py # Public API
|
| 1022 |
+
│ ├── compressor.py # ImageCompressor class
|
| 1023 |
+
│ └── trained_router.py # TrainedRouter (HuggingFace model)
|
| 1024 |
+
├── proxy/
|
| 1025 |
+
│ └── server.py # Integration point
|
| 1026 |
+
```
|
| 1027 |
+
|
| 1028 |
+
---
|
| 1029 |
+
|
| 1030 |
## File Structure Explained
|
| 1031 |
|
| 1032 |
```
|
docs/README.md
CHANGED
|
@@ -23,6 +23,7 @@ Welcome to the Headroom documentation.
|
|
| 23 |
| Topic | Description |
|
| 24 |
|-------|-------------|
|
| 25 |
| [Universal Compression](compression.md) | ML-based content detection + structure preservation |
|
|
|
|
| 26 |
| [Transforms](transforms.md) | How compression works |
|
| 27 |
| [CCR](ccr.md) | Reversible compression architecture |
|
| 28 |
| [Configuration](configuration.md) | All configuration options |
|
|
|
|
| 23 |
| Topic | Description |
|
| 24 |
|-------|-------------|
|
| 25 |
| [Universal Compression](compression.md) | ML-based content detection + structure preservation |
|
| 26 |
+
| [Image Compression](image-compression.md) | 40-90% token reduction for images via trained ML router |
|
| 27 |
| [Transforms](transforms.md) | How compression works |
|
| 28 |
| [CCR](ccr.md) | Reversible compression architecture |
|
| 29 |
| [Configuration](configuration.md) | All configuration options |
|
docs/image-compression.md
ADDED
|
@@ -0,0 +1,318 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Image Compression
|
| 2 |
+
|
| 3 |
+
Headroom automatically compresses images in your LLM requests, reducing token usage by **40-90%** while maintaining answer accuracy.
|
| 4 |
+
|
| 5 |
+
## Overview
|
| 6 |
+
|
| 7 |
+
Vision models charge by the token, and images are expensive:
|
| 8 |
+
- A 1024x1024 image costs ~765 tokens (OpenAI)
|
| 9 |
+
- A 2048x2048 image costs ~2,900 tokens
|
| 10 |
+
|
| 11 |
+
Headroom's image compression uses a **trained ML router** to analyze your query and automatically select the optimal compression technique:
|
| 12 |
+
|
| 13 |
+
| Technique | Savings | When Used |
|
| 14 |
+
|-----------|---------|-----------|
|
| 15 |
+
| `full_low` | ~87% | General questions ("What is this?") |
|
| 16 |
+
| `preserve` | 0% | Fine details needed ("Count the whiskers") |
|
| 17 |
+
| `crop` | 50-90% | Region-specific ("What's in the corner?") |
|
| 18 |
+
| `transcode` | ~99% | Text extraction ("Read the sign") |
|
| 19 |
+
|
| 20 |
+
## How It Works
|
| 21 |
+
|
| 22 |
+
```
|
| 23 |
+
User uploads image + asks question
|
| 24 |
+
↓
|
| 25 |
+
[Query Analysis]
|
| 26 |
+
TrainedRouter (MiniLM from HuggingFace)
|
| 27 |
+
Classifies: "What animal is this?" → full_low
|
| 28 |
+
↓
|
| 29 |
+
[Image Analysis]
|
| 30 |
+
SigLIP analyzes image properties
|
| 31 |
+
(has text? complex? fine details?)
|
| 32 |
+
↓
|
| 33 |
+
[Apply Compression]
|
| 34 |
+
OpenAI: detail="low"
|
| 35 |
+
Anthropic: Resize to 512px
|
| 36 |
+
Google: Resize to 768px
|
| 37 |
+
↓
|
| 38 |
+
Compressed request to LLM
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
## Quick Start
|
| 42 |
+
|
| 43 |
+
### With Headroom Proxy (Zero Code Changes)
|
| 44 |
+
|
| 45 |
+
```bash
|
| 46 |
+
# Start the proxy
|
| 47 |
+
headroom proxy --port 8787
|
| 48 |
+
|
| 49 |
+
# Connect your client
|
| 50 |
+
ANTHROPIC_BASE_URL=http://localhost:8787 claude
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
Images are automatically compressed based on your queries.
|
| 54 |
+
|
| 55 |
+
### With HeadroomClient
|
| 56 |
+
|
| 57 |
+
```python
|
| 58 |
+
from headroom import HeadroomClient
|
| 59 |
+
|
| 60 |
+
client = HeadroomClient(provider="openai")
|
| 61 |
+
|
| 62 |
+
response = client.chat.completions.create(
|
| 63 |
+
model="gpt-4o",
|
| 64 |
+
messages=[{
|
| 65 |
+
"role": "user",
|
| 66 |
+
"content": [
|
| 67 |
+
{"type": "text", "text": "What animal is this?"},
|
| 68 |
+
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
|
| 69 |
+
]
|
| 70 |
+
}]
|
| 71 |
+
)
|
| 72 |
+
# Image automatically compressed with detail="low" (87% savings)
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
### Direct API
|
| 76 |
+
|
| 77 |
+
```python
|
| 78 |
+
from headroom.image import ImageCompressor
|
| 79 |
+
|
| 80 |
+
compressor = ImageCompressor()
|
| 81 |
+
|
| 82 |
+
# Compress images in messages
|
| 83 |
+
compressed_messages = compressor.compress(messages, provider="openai")
|
| 84 |
+
|
| 85 |
+
# Check savings
|
| 86 |
+
print(f"Saved {compressor.last_savings:.0f}% tokens")
|
| 87 |
+
print(f"Technique: {compressor.last_result.technique.value}")
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
## Configuration
|
| 91 |
+
|
| 92 |
+
### Proxy Configuration
|
| 93 |
+
|
| 94 |
+
```bash
|
| 95 |
+
# Enable image compression (default: true)
|
| 96 |
+
headroom proxy --image-optimize
|
| 97 |
+
|
| 98 |
+
# Disable image compression
|
| 99 |
+
headroom proxy --no-image-optimize
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
### Programmatic Configuration
|
| 103 |
+
|
| 104 |
+
```python
|
| 105 |
+
from headroom.image import ImageCompressor
|
| 106 |
+
|
| 107 |
+
compressor = ImageCompressor(
|
| 108 |
+
model_id="chopratejas/technique-router", # HuggingFace model
|
| 109 |
+
use_siglip=True, # Enable image analysis
|
| 110 |
+
device="cuda", # Use GPU if available
|
| 111 |
+
)
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
## Provider Support
|
| 115 |
+
|
| 116 |
+
| Provider | Detection | Compression Method |
|
| 117 |
+
|----------|-----------|-------------------|
|
| 118 |
+
| **OpenAI** | `image_url` | Sets `detail="low"` |
|
| 119 |
+
| **Anthropic** | `image` with `source` | Resizes to 512px |
|
| 120 |
+
| **Google** | `inlineData` | Resizes to 768px (tile-optimized) |
|
| 121 |
+
|
| 122 |
+
### OpenAI
|
| 123 |
+
|
| 124 |
+
Uses the native `detail` parameter:
|
| 125 |
+
```python
|
| 126 |
+
# Before
|
| 127 |
+
{"type": "image_url", "image_url": {"url": "data:..."}}
|
| 128 |
+
|
| 129 |
+
# After (full_low technique)
|
| 130 |
+
{"type": "image_url", "image_url": {"url": "data:...", "detail": "low"}}
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
### Anthropic
|
| 134 |
+
|
| 135 |
+
Resizes the image using PIL:
|
| 136 |
+
```python
|
| 137 |
+
# Before: 1024x1024 image (~1,398 tokens)
|
| 138 |
+
# After: 512x512 image (~349 tokens) - 75% savings
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
### Google Gemini
|
| 142 |
+
|
| 143 |
+
Resizes to 768px (optimal for Gemini's 768x768 tile system):
|
| 144 |
+
```python
|
| 145 |
+
# Before: 1536x1536 image (4 tiles × 258 = 1,032 tokens)
|
| 146 |
+
# After: 768x768 image (1 tile × 258 = 258 tokens) - 75% savings
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
+
## Techniques Explained
|
| 150 |
+
|
| 151 |
+
### `full_low` (87% savings)
|
| 152 |
+
|
| 153 |
+
Best for general understanding questions:
|
| 154 |
+
- "What is this?"
|
| 155 |
+
- "Describe the scene"
|
| 156 |
+
- "Is this indoors or outdoors?"
|
| 157 |
+
|
| 158 |
+
The model doesn't need fine details to answer these questions.
|
| 159 |
+
|
| 160 |
+
### `preserve` (0% savings)
|
| 161 |
+
|
| 162 |
+
Required when fine details matter:
|
| 163 |
+
- "Count the whiskers"
|
| 164 |
+
- "What brand is shown?"
|
| 165 |
+
- "Read the serial number"
|
| 166 |
+
- "What time does the clock show?"
|
| 167 |
+
|
| 168 |
+
### `crop` (50-90% savings)
|
| 169 |
+
|
| 170 |
+
For region-specific queries:
|
| 171 |
+
- "What's in the top-right corner?"
|
| 172 |
+
- "Focus on the background"
|
| 173 |
+
- "Zoom into the left side"
|
| 174 |
+
|
| 175 |
+
*Note: Currently implemented as resize. True cropping coming soon.*
|
| 176 |
+
|
| 177 |
+
### `transcode` (99% savings)
|
| 178 |
+
|
| 179 |
+
For text extraction (converts image to text):
|
| 180 |
+
- "Read the sign"
|
| 181 |
+
- "What does it say?"
|
| 182 |
+
- "Transcribe the document"
|
| 183 |
+
|
| 184 |
+
*Note: Requires vision model call. Currently falls back to preserve.*
|
| 185 |
+
|
| 186 |
+
## The Trained Router
|
| 187 |
+
|
| 188 |
+
The routing decision is made by a fine-tuned **MiniLM** classifier:
|
| 189 |
+
|
| 190 |
+
- **Model**: `chopratejas/technique-router` on HuggingFace
|
| 191 |
+
- **Size**: ~128MB
|
| 192 |
+
- **Accuracy**: 93.7% on validation set
|
| 193 |
+
- **Training data**: 1,157 examples across 4 techniques
|
| 194 |
+
|
| 195 |
+
The model is downloaded automatically on first use and cached locally.
|
| 196 |
+
|
| 197 |
+
### Training Data Examples
|
| 198 |
+
|
| 199 |
+
| Query | Technique |
|
| 200 |
+
|-------|-----------|
|
| 201 |
+
| "What animal is this?" | `full_low` |
|
| 202 |
+
| "Count the spots" | `preserve` |
|
| 203 |
+
| "Read the text on the sign" | `transcode` |
|
| 204 |
+
| "What's in the corner?" | `crop` |
|
| 205 |
+
|
| 206 |
+
## Performance
|
| 207 |
+
|
| 208 |
+
### Token Savings by Query Type
|
| 209 |
+
|
| 210 |
+
| Query Type | Before | After | Savings |
|
| 211 |
+
|------------|--------|-------|---------|
|
| 212 |
+
| General ("What is this?") | 765 | 85 | 89% |
|
| 213 |
+
| Detail ("Count items") | 765 | 765 | 0% |
|
| 214 |
+
| Region ("Top corner?") | 765 | 85 | 89% |
|
| 215 |
+
| Text ("Read the sign") | 765 | 85 | 89% |
|
| 216 |
+
|
| 217 |
+
### Latency
|
| 218 |
+
|
| 219 |
+
- Router inference: ~10ms (CPU), ~2ms (GPU)
|
| 220 |
+
- Image resize: ~5-20ms depending on size
|
| 221 |
+
- First request: +2-3s (model download, cached after)
|
| 222 |
+
|
| 223 |
+
## Troubleshooting
|
| 224 |
+
|
| 225 |
+
### Model Download Issues
|
| 226 |
+
|
| 227 |
+
The HuggingFace model downloads on first use:
|
| 228 |
+
|
| 229 |
+
```python
|
| 230 |
+
# Force a specific cache directory
|
| 231 |
+
import os
|
| 232 |
+
os.environ["HF_HOME"] = "/path/to/cache"
|
| 233 |
+
|
| 234 |
+
from headroom.image import ImageCompressor
|
| 235 |
+
compressor = ImageCompressor()
|
| 236 |
+
```
|
| 237 |
+
|
| 238 |
+
### GPU Memory
|
| 239 |
+
|
| 240 |
+
SigLIP requires ~400MB GPU memory. To use CPU only:
|
| 241 |
+
|
| 242 |
+
```python
|
| 243 |
+
compressor = ImageCompressor(device="cpu")
|
| 244 |
+
```
|
| 245 |
+
|
| 246 |
+
### Disable Image Compression
|
| 247 |
+
|
| 248 |
+
```python
|
| 249 |
+
# Proxy
|
| 250 |
+
headroom proxy --no-image-optimize
|
| 251 |
+
|
| 252 |
+
# Direct
|
| 253 |
+
# Simply don't call compress()
|
| 254 |
+
```
|
| 255 |
+
|
| 256 |
+
## API Reference
|
| 257 |
+
|
| 258 |
+
### `ImageCompressor`
|
| 259 |
+
|
| 260 |
+
```python
|
| 261 |
+
class ImageCompressor:
|
| 262 |
+
def __init__(
|
| 263 |
+
self,
|
| 264 |
+
model_id: str = "chopratejas/technique-router",
|
| 265 |
+
use_siglip: bool = True,
|
| 266 |
+
device: str | None = None,
|
| 267 |
+
): ...
|
| 268 |
+
|
| 269 |
+
def has_images(self, messages: list[dict]) -> bool:
|
| 270 |
+
"""Check if messages contain images."""
|
| 271 |
+
|
| 272 |
+
def compress(
|
| 273 |
+
self,
|
| 274 |
+
messages: list[dict],
|
| 275 |
+
provider: str = "openai",
|
| 276 |
+
) -> list[dict]:
|
| 277 |
+
"""Compress images in messages."""
|
| 278 |
+
|
| 279 |
+
@property
|
| 280 |
+
def last_result(self) -> CompressionResult | None:
|
| 281 |
+
"""Result of last compression."""
|
| 282 |
+
|
| 283 |
+
@property
|
| 284 |
+
def last_savings(self) -> float:
|
| 285 |
+
"""Savings percentage from last compression."""
|
| 286 |
+
```
|
| 287 |
+
|
| 288 |
+
### `CompressionResult`
|
| 289 |
+
|
| 290 |
+
```python
|
| 291 |
+
@dataclass
|
| 292 |
+
class CompressionResult:
|
| 293 |
+
technique: Technique # full_low, preserve, crop, transcode
|
| 294 |
+
original_tokens: int # Estimated tokens before
|
| 295 |
+
compressed_tokens: int # Estimated tokens after
|
| 296 |
+
confidence: float # Router confidence (0-1)
|
| 297 |
+
|
| 298 |
+
@property
|
| 299 |
+
def savings_percent(self) -> float:
|
| 300 |
+
"""Percentage of tokens saved."""
|
| 301 |
+
```
|
| 302 |
+
|
| 303 |
+
### `Technique`
|
| 304 |
+
|
| 305 |
+
```python
|
| 306 |
+
class Technique(Enum):
|
| 307 |
+
FULL_LOW = "full_low" # 87% savings
|
| 308 |
+
PRESERVE = "preserve" # 0% savings
|
| 309 |
+
CROP = "crop" # 50-90% savings
|
| 310 |
+
TRANSCODE = "transcode" # 99% savings
|
| 311 |
+
```
|
| 312 |
+
|
| 313 |
+
## See Also
|
| 314 |
+
|
| 315 |
+
- [Compression Guide](compression.md) - Text compression techniques
|
| 316 |
+
- [CCR Guide](ccr.md) - Reversible compression with retrieval
|
| 317 |
+
- [Proxy Guide](proxy.md) - Zero-code deployment
|
| 318 |
+
- [Architecture](ARCHITECTURE.md) - System design
|
headroom/image/__init__.py
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Image token compression for Headroom.
|
| 2 |
+
|
| 3 |
+
Automatically compress images in LLM requests to save 40-90% tokens
|
| 4 |
+
while maintaining answer accuracy.
|
| 5 |
+
|
| 6 |
+
Usage:
|
| 7 |
+
from headroom.image import ImageCompressor
|
| 8 |
+
|
| 9 |
+
compressor = ImageCompressor()
|
| 10 |
+
|
| 11 |
+
# Check if messages have images
|
| 12 |
+
if compressor.has_images(messages):
|
| 13 |
+
# Compress based on query intent
|
| 14 |
+
messages = compressor.compress(messages, provider="openai")
|
| 15 |
+
print(f"Saved {compressor.last_savings:.0f}% tokens")
|
| 16 |
+
|
| 17 |
+
Or use the convenience function:
|
| 18 |
+
from headroom.image import compress_images
|
| 19 |
+
|
| 20 |
+
messages = compress_images(messages, provider="openai")
|
| 21 |
+
|
| 22 |
+
The compression technique is selected by a trained ML model:
|
| 23 |
+
- FULL_LOW: General questions → 87% savings (detail="low")
|
| 24 |
+
- PRESERVE: Fine details needed → 0% savings (keep quality)
|
| 25 |
+
- CROP: Region-specific → 50-90% savings (extract region)
|
| 26 |
+
- TRANSCODE: Text extraction → 99% savings (OCR to text)
|
| 27 |
+
|
| 28 |
+
Model: https://huggingface.co/chopratejas/technique-router
|
| 29 |
+
"""
|
| 30 |
+
|
| 31 |
+
from .compressor import (
|
| 32 |
+
CompressionResult,
|
| 33 |
+
ImageCompressor,
|
| 34 |
+
Technique,
|
| 35 |
+
compress_images,
|
| 36 |
+
get_compressor,
|
| 37 |
+
)
|
| 38 |
+
|
| 39 |
+
__all__ = [
|
| 40 |
+
# Main API
|
| 41 |
+
"ImageCompressor",
|
| 42 |
+
"compress_images",
|
| 43 |
+
"get_compressor",
|
| 44 |
+
# Types
|
| 45 |
+
"Technique",
|
| 46 |
+
"CompressionResult",
|
| 47 |
+
]
|
headroom/image/compressor.py
ADDED
|
@@ -0,0 +1,450 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Image Compressor - Seamless image token optimization.
|
| 2 |
+
|
| 3 |
+
This is the main entry point for image compression in Headroom.
|
| 4 |
+
It automatically:
|
| 5 |
+
1. Detects images in messages
|
| 6 |
+
2. Extracts the user's query
|
| 7 |
+
3. Routes to optimal compression technique (via trained model)
|
| 8 |
+
4. Applies provider-specific compression
|
| 9 |
+
|
| 10 |
+
Usage:
|
| 11 |
+
from headroom.image import ImageCompressor
|
| 12 |
+
|
| 13 |
+
compressor = ImageCompressor()
|
| 14 |
+
|
| 15 |
+
# Compress images in a request
|
| 16 |
+
compressed = compressor.compress(messages, provider="openai")
|
| 17 |
+
|
| 18 |
+
# Check savings
|
| 19 |
+
print(f"Saved {compressor.last_savings}% tokens")
|
| 20 |
+
"""
|
| 21 |
+
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
|
| 24 |
+
import base64
|
| 25 |
+
import io
|
| 26 |
+
import logging
|
| 27 |
+
import re
|
| 28 |
+
from dataclasses import dataclass
|
| 29 |
+
from typing import TYPE_CHECKING, Any
|
| 30 |
+
|
| 31 |
+
if TYPE_CHECKING:
|
| 32 |
+
from .trained_router import TrainedRouter
|
| 33 |
+
|
| 34 |
+
from .trained_router import Technique
|
| 35 |
+
|
| 36 |
+
logger = logging.getLogger(__name__)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
@dataclass
|
| 40 |
+
class CompressionResult:
|
| 41 |
+
"""Result of image compression."""
|
| 42 |
+
|
| 43 |
+
technique: Technique
|
| 44 |
+
original_tokens: int
|
| 45 |
+
compressed_tokens: int
|
| 46 |
+
confidence: float
|
| 47 |
+
|
| 48 |
+
@property
|
| 49 |
+
def savings_percent(self) -> float:
|
| 50 |
+
if self.original_tokens == 0:
|
| 51 |
+
return 0.0
|
| 52 |
+
return (1 - self.compressed_tokens / self.original_tokens) * 100
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
class ImageCompressor:
|
| 56 |
+
"""Seamless image compression for LLM requests.
|
| 57 |
+
|
| 58 |
+
Automatically detects images, analyzes queries, and applies
|
| 59 |
+
optimal compression based on a trained ML model.
|
| 60 |
+
|
| 61 |
+
The model is downloaded from HuggingFace on first use:
|
| 62 |
+
https://huggingface.co/chopratejas/technique-router
|
| 63 |
+
|
| 64 |
+
Args:
|
| 65 |
+
model_id: HuggingFace model ID (default: chopratejas/technique-router)
|
| 66 |
+
use_siglip: Whether to use SigLIP for image analysis (default: True)
|
| 67 |
+
device: Device for inference ('cuda', 'cpu', or None for auto)
|
| 68 |
+
"""
|
| 69 |
+
|
| 70 |
+
DEFAULT_MODEL = "chopratejas/technique-router"
|
| 71 |
+
|
| 72 |
+
def __init__(
|
| 73 |
+
self,
|
| 74 |
+
model_id: str = DEFAULT_MODEL,
|
| 75 |
+
use_siglip: bool = True,
|
| 76 |
+
device: str | None = None,
|
| 77 |
+
):
|
| 78 |
+
self.model_id = model_id
|
| 79 |
+
self.use_siglip = use_siglip
|
| 80 |
+
self.device = device
|
| 81 |
+
|
| 82 |
+
# Lazy-loaded router
|
| 83 |
+
self._router: TrainedRouter | None = None
|
| 84 |
+
|
| 85 |
+
# Last compression result (for metrics)
|
| 86 |
+
self.last_result: CompressionResult | None = None
|
| 87 |
+
|
| 88 |
+
@property
|
| 89 |
+
def last_savings(self) -> float:
|
| 90 |
+
"""Savings from last compression (percentage)."""
|
| 91 |
+
if self.last_result:
|
| 92 |
+
return self.last_result.savings_percent
|
| 93 |
+
return 0.0
|
| 94 |
+
|
| 95 |
+
def _get_router(self) -> TrainedRouter:
|
| 96 |
+
"""Lazy load the trained router."""
|
| 97 |
+
if self._router is None:
|
| 98 |
+
from .trained_router import TrainedRouter
|
| 99 |
+
|
| 100 |
+
self._router = TrainedRouter(
|
| 101 |
+
model_path=self.model_id,
|
| 102 |
+
use_siglip=self.use_siglip,
|
| 103 |
+
device=self.device,
|
| 104 |
+
)
|
| 105 |
+
return self._router
|
| 106 |
+
|
| 107 |
+
def has_images(self, messages: list[dict[str, Any]]) -> bool:
|
| 108 |
+
"""Check if messages contain images."""
|
| 109 |
+
for message in messages:
|
| 110 |
+
content = message.get("content")
|
| 111 |
+
if isinstance(content, list):
|
| 112 |
+
for item in content:
|
| 113 |
+
if isinstance(item, dict):
|
| 114 |
+
# OpenAI format
|
| 115 |
+
if item.get("type") == "image_url":
|
| 116 |
+
return True
|
| 117 |
+
# Anthropic format
|
| 118 |
+
if item.get("type") == "image":
|
| 119 |
+
return True
|
| 120 |
+
# Google format
|
| 121 |
+
if "inlineData" in item:
|
| 122 |
+
return True
|
| 123 |
+
return False
|
| 124 |
+
|
| 125 |
+
def _extract_query(self, messages: list[dict[str, Any]]) -> str:
|
| 126 |
+
"""Extract the text query from messages."""
|
| 127 |
+
# Look for user message with text
|
| 128 |
+
for message in reversed(messages):
|
| 129 |
+
if message.get("role") != "user":
|
| 130 |
+
continue
|
| 131 |
+
|
| 132 |
+
content = message.get("content")
|
| 133 |
+
|
| 134 |
+
# Simple string content
|
| 135 |
+
if isinstance(content, str):
|
| 136 |
+
return content
|
| 137 |
+
|
| 138 |
+
# Multi-part content
|
| 139 |
+
if isinstance(content, list):
|
| 140 |
+
texts = []
|
| 141 |
+
for item in content:
|
| 142 |
+
if isinstance(item, dict):
|
| 143 |
+
if item.get("type") == "text":
|
| 144 |
+
texts.append(item.get("text", ""))
|
| 145 |
+
elif isinstance(item, str):
|
| 146 |
+
texts.append(item)
|
| 147 |
+
if texts:
|
| 148 |
+
return " ".join(texts)
|
| 149 |
+
|
| 150 |
+
return ""
|
| 151 |
+
|
| 152 |
+
def _extract_image_data(self, messages: list[dict[str, Any]]) -> bytes | None:
|
| 153 |
+
"""Extract first image data from messages."""
|
| 154 |
+
for message in messages:
|
| 155 |
+
content = message.get("content")
|
| 156 |
+
if not isinstance(content, list):
|
| 157 |
+
continue
|
| 158 |
+
|
| 159 |
+
for item in content:
|
| 160 |
+
if not isinstance(item, dict):
|
| 161 |
+
continue
|
| 162 |
+
|
| 163 |
+
# OpenAI format: {"type": "image_url", "image_url": {"url": "data:..."}}
|
| 164 |
+
if item.get("type") == "image_url":
|
| 165 |
+
url = item.get("image_url", {}).get("url", "")
|
| 166 |
+
if url.startswith("data:"):
|
| 167 |
+
# Extract base64 data
|
| 168 |
+
match = re.match(r"data:image/[^;]+;base64,(.+)", url)
|
| 169 |
+
if match:
|
| 170 |
+
return base64.b64decode(match.group(1))
|
| 171 |
+
|
| 172 |
+
# Anthropic format: {"type": "image", "source": {"data": "..."}}
|
| 173 |
+
if item.get("type") == "image":
|
| 174 |
+
source = item.get("source", {})
|
| 175 |
+
if source.get("type") == "base64":
|
| 176 |
+
return base64.b64decode(source.get("data", ""))
|
| 177 |
+
|
| 178 |
+
# Google format: {"inlineData": {"data": "..."}}
|
| 179 |
+
if "inlineData" in item:
|
| 180 |
+
return base64.b64decode(item["inlineData"].get("data", ""))
|
| 181 |
+
|
| 182 |
+
return None
|
| 183 |
+
|
| 184 |
+
def _resize_image(
|
| 185 |
+
self, image_data: bytes, max_dimension: int = 512, quality: int = 85
|
| 186 |
+
) -> tuple[bytes, str]:
|
| 187 |
+
"""Resize image to reduce tokens.
|
| 188 |
+
|
| 189 |
+
Args:
|
| 190 |
+
image_data: Original image bytes
|
| 191 |
+
max_dimension: Maximum width or height
|
| 192 |
+
quality: JPEG quality (1-100)
|
| 193 |
+
|
| 194 |
+
Returns:
|
| 195 |
+
Tuple of (resized_bytes, media_type)
|
| 196 |
+
"""
|
| 197 |
+
from PIL import Image
|
| 198 |
+
|
| 199 |
+
img = Image.open(io.BytesIO(image_data))
|
| 200 |
+
original_format = img.format or "PNG"
|
| 201 |
+
|
| 202 |
+
# Calculate new dimensions preserving aspect ratio
|
| 203 |
+
width, height = img.size
|
| 204 |
+
if width <= max_dimension and height <= max_dimension:
|
| 205 |
+
# Already small enough
|
| 206 |
+
return image_data, f"image/{original_format.lower()}"
|
| 207 |
+
|
| 208 |
+
if width > height:
|
| 209 |
+
new_width = max_dimension
|
| 210 |
+
new_height = int(height * (max_dimension / width))
|
| 211 |
+
else:
|
| 212 |
+
new_height = max_dimension
|
| 213 |
+
new_width = int(width * (max_dimension / height))
|
| 214 |
+
|
| 215 |
+
# Resize
|
| 216 |
+
resized = img.resize((new_width, new_height), Image.Resampling.LANCZOS)
|
| 217 |
+
|
| 218 |
+
# Convert to RGB if needed (for JPEG)
|
| 219 |
+
if resized.mode in ("RGBA", "P"):
|
| 220 |
+
resized = resized.convert("RGB")
|
| 221 |
+
|
| 222 |
+
# Save as JPEG for best compression
|
| 223 |
+
buf = io.BytesIO()
|
| 224 |
+
resized.save(buf, format="JPEG", quality=quality, optimize=True)
|
| 225 |
+
return buf.getvalue(), "image/jpeg"
|
| 226 |
+
|
| 227 |
+
def _estimate_tokens(self, image_data: bytes, detail: str = "high") -> int:
|
| 228 |
+
"""Estimate token count for image (OpenAI formula)."""
|
| 229 |
+
try:
|
| 230 |
+
from PIL import Image
|
| 231 |
+
|
| 232 |
+
img = Image.open(io.BytesIO(image_data))
|
| 233 |
+
width, height = img.size
|
| 234 |
+
except Exception:
|
| 235 |
+
# Default estimate
|
| 236 |
+
return 765
|
| 237 |
+
|
| 238 |
+
if detail == "low":
|
| 239 |
+
return 85
|
| 240 |
+
|
| 241 |
+
# High detail: 85 tokens per 512x512 tile + 170 base
|
| 242 |
+
tiles_x = (width + 511) // 512
|
| 243 |
+
tiles_y = (height + 511) // 512
|
| 244 |
+
return 85 * tiles_x * tiles_y + 170
|
| 245 |
+
|
| 246 |
+
def _apply_compression(
|
| 247 |
+
self,
|
| 248 |
+
messages: list[dict[str, Any]],
|
| 249 |
+
technique: Technique,
|
| 250 |
+
provider: str,
|
| 251 |
+
) -> list[dict[str, Any]]:
|
| 252 |
+
"""Apply compression technique to messages."""
|
| 253 |
+
if technique.value == "preserve":
|
| 254 |
+
return messages
|
| 255 |
+
|
| 256 |
+
compressed = []
|
| 257 |
+
for message in messages:
|
| 258 |
+
content = message.get("content")
|
| 259 |
+
|
| 260 |
+
if not isinstance(content, list):
|
| 261 |
+
compressed.append(message)
|
| 262 |
+
continue
|
| 263 |
+
|
| 264 |
+
new_content = []
|
| 265 |
+
for item in content:
|
| 266 |
+
if not isinstance(item, dict):
|
| 267 |
+
new_content.append(item)
|
| 268 |
+
continue
|
| 269 |
+
|
| 270 |
+
# OpenAI format - compare by value since technique may be from trained_router
|
| 271 |
+
if item.get("type") == "image_url" and provider == "openai":
|
| 272 |
+
if technique.value == "full_low":
|
| 273 |
+
# Apply detail="low"
|
| 274 |
+
new_item = {
|
| 275 |
+
"type": "image_url",
|
| 276 |
+
"image_url": {
|
| 277 |
+
**item.get("image_url", {}),
|
| 278 |
+
"detail": "low",
|
| 279 |
+
},
|
| 280 |
+
}
|
| 281 |
+
new_content.append(new_item)
|
| 282 |
+
elif technique.value == "crop":
|
| 283 |
+
# For now, use low detail (TODO: implement actual cropping)
|
| 284 |
+
new_item = {
|
| 285 |
+
"type": "image_url",
|
| 286 |
+
"image_url": {
|
| 287 |
+
**item.get("image_url", {}),
|
| 288 |
+
"detail": "low",
|
| 289 |
+
},
|
| 290 |
+
}
|
| 291 |
+
new_content.append(new_item)
|
| 292 |
+
elif technique.value == "transcode":
|
| 293 |
+
# TODO: Convert to text description
|
| 294 |
+
# For now, keep original
|
| 295 |
+
new_content.append(item)
|
| 296 |
+
else:
|
| 297 |
+
new_content.append(item)
|
| 298 |
+
|
| 299 |
+
# Anthropic format - resize image for compression
|
| 300 |
+
elif item.get("type") == "image" and provider == "anthropic":
|
| 301 |
+
if technique.value in ("full_low", "crop"):
|
| 302 |
+
# Resize image to reduce tokens
|
| 303 |
+
try:
|
| 304 |
+
source = item.get("source", {})
|
| 305 |
+
if source.get("type") == "base64":
|
| 306 |
+
original_data = base64.b64decode(source.get("data", ""))
|
| 307 |
+
resized_data, media_type = self._resize_image(
|
| 308 |
+
original_data, max_dimension=512
|
| 309 |
+
)
|
| 310 |
+
new_item = {
|
| 311 |
+
"type": "image",
|
| 312 |
+
"source": {
|
| 313 |
+
"type": "base64",
|
| 314 |
+
"media_type": media_type,
|
| 315 |
+
"data": base64.b64encode(resized_data).decode(),
|
| 316 |
+
},
|
| 317 |
+
}
|
| 318 |
+
new_content.append(new_item)
|
| 319 |
+
else:
|
| 320 |
+
new_content.append(item)
|
| 321 |
+
except Exception as e:
|
| 322 |
+
logger.warning(f"Failed to resize Anthropic image: {e}")
|
| 323 |
+
new_content.append(item)
|
| 324 |
+
else:
|
| 325 |
+
new_content.append(item)
|
| 326 |
+
|
| 327 |
+
# Google format - resize image for compression
|
| 328 |
+
elif "inlineData" in item and provider == "google":
|
| 329 |
+
if technique.value in ("full_low", "crop"):
|
| 330 |
+
try:
|
| 331 |
+
inline = item.get("inlineData", {})
|
| 332 |
+
original_data = base64.b64decode(inline.get("data", ""))
|
| 333 |
+
resized_data, media_type = self._resize_image(
|
| 334 |
+
original_data,
|
| 335 |
+
max_dimension=768, # Google uses 768x768 tiles
|
| 336 |
+
)
|
| 337 |
+
new_item = {
|
| 338 |
+
"inlineData": {
|
| 339 |
+
"mimeType": media_type,
|
| 340 |
+
"data": base64.b64encode(resized_data).decode(),
|
| 341 |
+
}
|
| 342 |
+
}
|
| 343 |
+
new_content.append(new_item)
|
| 344 |
+
except Exception as e:
|
| 345 |
+
logger.warning(f"Failed to resize Google image: {e}")
|
| 346 |
+
new_content.append(item)
|
| 347 |
+
else:
|
| 348 |
+
new_content.append(item)
|
| 349 |
+
|
| 350 |
+
else:
|
| 351 |
+
new_content.append(item)
|
| 352 |
+
|
| 353 |
+
compressed.append({**message, "content": new_content})
|
| 354 |
+
|
| 355 |
+
return compressed
|
| 356 |
+
|
| 357 |
+
def compress(
|
| 358 |
+
self,
|
| 359 |
+
messages: list[dict[str, Any]],
|
| 360 |
+
provider: str = "openai",
|
| 361 |
+
) -> list[dict[str, Any]]:
|
| 362 |
+
"""Compress images in messages.
|
| 363 |
+
|
| 364 |
+
Args:
|
| 365 |
+
messages: LLM messages (OpenAI/Anthropic/Google format)
|
| 366 |
+
provider: Target provider ('openai', 'anthropic', 'google')
|
| 367 |
+
|
| 368 |
+
Returns:
|
| 369 |
+
Messages with compressed images
|
| 370 |
+
"""
|
| 371 |
+
if not self.has_images(messages):
|
| 372 |
+
return messages
|
| 373 |
+
|
| 374 |
+
# Extract query and image
|
| 375 |
+
query = self._extract_query(messages)
|
| 376 |
+
image_data = self._extract_image_data(messages)
|
| 377 |
+
|
| 378 |
+
if not query or not image_data:
|
| 379 |
+
logger.debug("Could not extract query or image, skipping compression")
|
| 380 |
+
return messages
|
| 381 |
+
|
| 382 |
+
# Route to technique
|
| 383 |
+
try:
|
| 384 |
+
router = self._get_router()
|
| 385 |
+
decision = router.classify(image_data, query)
|
| 386 |
+
technique = decision.technique
|
| 387 |
+
confidence = decision.confidence
|
| 388 |
+
except Exception as e:
|
| 389 |
+
logger.warning(f"Router failed, preserving image: {e}")
|
| 390 |
+
technique = Technique.PRESERVE
|
| 391 |
+
confidence = 0.0
|
| 392 |
+
|
| 393 |
+
# Calculate tokens - compare by value since technique is from trained_router
|
| 394 |
+
original_tokens = self._estimate_tokens(image_data, "high")
|
| 395 |
+
|
| 396 |
+
if technique.value == "full_low":
|
| 397 |
+
compressed_tokens = 85 # OpenAI low detail
|
| 398 |
+
elif technique.value == "preserve":
|
| 399 |
+
compressed_tokens = original_tokens
|
| 400 |
+
elif technique.value == "crop":
|
| 401 |
+
compressed_tokens = 85 # Approximation
|
| 402 |
+
elif technique.value == "transcode":
|
| 403 |
+
compressed_tokens = 50 # Text description estimate
|
| 404 |
+
else:
|
| 405 |
+
compressed_tokens = original_tokens
|
| 406 |
+
|
| 407 |
+
# Store result
|
| 408 |
+
self.last_result = CompressionResult(
|
| 409 |
+
technique=technique,
|
| 410 |
+
original_tokens=original_tokens,
|
| 411 |
+
compressed_tokens=compressed_tokens,
|
| 412 |
+
confidence=confidence,
|
| 413 |
+
)
|
| 414 |
+
|
| 415 |
+
logger.info(
|
| 416 |
+
f"Image compression: {technique.value} "
|
| 417 |
+
f"({original_tokens} → {compressed_tokens} tokens, "
|
| 418 |
+
f"{self.last_result.savings_percent:.0f}% saved)"
|
| 419 |
+
)
|
| 420 |
+
|
| 421 |
+
# Apply compression
|
| 422 |
+
return self._apply_compression(messages, technique, provider)
|
| 423 |
+
|
| 424 |
+
|
| 425 |
+
# Singleton for convenience
|
| 426 |
+
_default_compressor: ImageCompressor | None = None
|
| 427 |
+
|
| 428 |
+
|
| 429 |
+
def get_compressor() -> ImageCompressor:
|
| 430 |
+
"""Get the default ImageCompressor instance."""
|
| 431 |
+
global _default_compressor
|
| 432 |
+
if _default_compressor is None:
|
| 433 |
+
_default_compressor = ImageCompressor()
|
| 434 |
+
return _default_compressor
|
| 435 |
+
|
| 436 |
+
|
| 437 |
+
def compress_images(
|
| 438 |
+
messages: list[dict[str, Any]],
|
| 439 |
+
provider: str = "openai",
|
| 440 |
+
) -> list[dict[str, Any]]:
|
| 441 |
+
"""Convenience function to compress images in messages.
|
| 442 |
+
|
| 443 |
+
Args:
|
| 444 |
+
messages: LLM messages
|
| 445 |
+
provider: Target provider
|
| 446 |
+
|
| 447 |
+
Returns:
|
| 448 |
+
Messages with compressed images
|
| 449 |
+
"""
|
| 450 |
+
return get_compressor().compress(messages, provider)
|
headroom/image/trained_router.py
ADDED
|
@@ -0,0 +1,328 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Trained Technique Router using fine-tuned MiniLM + SigLIP.
|
| 2 |
+
|
| 3 |
+
Uses a TRAINED classifier for query intent:
|
| 4 |
+
1. MiniLM classifier: Fine-tuned on 1157 examples (93.7% accuracy)
|
| 5 |
+
2. SigLIP: Analyzes image properties
|
| 6 |
+
3. Combined decision based on both signals
|
| 7 |
+
|
| 8 |
+
The MiniLM model is hosted on HuggingFace: headroom-ai/technique-router
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import io
|
| 14 |
+
from dataclasses import dataclass
|
| 15 |
+
from enum import Enum
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
|
| 18 |
+
import torch
|
| 19 |
+
from PIL import Image
|
| 20 |
+
from transformers import (
|
| 21 |
+
AutoModel,
|
| 22 |
+
AutoModelForSequenceClassification,
|
| 23 |
+
AutoProcessor,
|
| 24 |
+
AutoTokenizer,
|
| 25 |
+
)
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
class Technique(Enum):
|
| 29 |
+
"""Image optimization techniques."""
|
| 30 |
+
|
| 31 |
+
TRANSCODE = "transcode" # Convert to text description (99% savings)
|
| 32 |
+
CROP = "crop" # Extract relevant region (50-90% savings)
|
| 33 |
+
PRESERVE = "preserve" # Keep full quality (0% savings)
|
| 34 |
+
FULL_LOW = "full_low" # Full image, lower quality (87% savings)
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
@dataclass
|
| 38 |
+
class ImageSignals:
|
| 39 |
+
"""Signals extracted from image analysis."""
|
| 40 |
+
|
| 41 |
+
has_text: float
|
| 42 |
+
is_document: float
|
| 43 |
+
is_complex: float
|
| 44 |
+
has_small_details: float
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
@dataclass
|
| 48 |
+
class RouteDecision:
|
| 49 |
+
"""Result of routing decision."""
|
| 50 |
+
|
| 51 |
+
technique: Technique
|
| 52 |
+
confidence: float
|
| 53 |
+
reason: str
|
| 54 |
+
image_signals: ImageSignals | None = None
|
| 55 |
+
query_prediction: str | None = None
|
| 56 |
+
query_confidence: float | None = None
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
class TrainedRouter:
|
| 60 |
+
"""Router using trained MiniLM classifier + SigLIP image analysis.
|
| 61 |
+
|
| 62 |
+
This router uses:
|
| 63 |
+
1. A fine-tuned MiniLM classifier for query intent (93.7% accuracy)
|
| 64 |
+
2. SigLIP for image property analysis
|
| 65 |
+
3. Combined decision logic
|
| 66 |
+
|
| 67 |
+
The MiniLM model can be loaded from:
|
| 68 |
+
- Local path (for development)
|
| 69 |
+
- HuggingFace Hub: headroom-ai/technique-router (for production)
|
| 70 |
+
"""
|
| 71 |
+
|
| 72 |
+
# Model identifiers
|
| 73 |
+
DEFAULT_HF_MODEL = "chopratejas/technique-router"
|
| 74 |
+
SIGLIP_MODEL = "google/siglip-base-patch16-224"
|
| 75 |
+
|
| 76 |
+
# Image analysis prompts for SigLIP
|
| 77 |
+
IMAGE_DESCRIPTIONS = {
|
| 78 |
+
"has_text": [
|
| 79 |
+
"an image with visible text, words, or writing",
|
| 80 |
+
"a sign, label, or document with readable text",
|
| 81 |
+
],
|
| 82 |
+
"is_document": [
|
| 83 |
+
"a document, form, receipt, or page with text",
|
| 84 |
+
"a scanned paper or screenshot of text",
|
| 85 |
+
],
|
| 86 |
+
"is_complex": [
|
| 87 |
+
"a complex scene with many objects and details",
|
| 88 |
+
"a cluttered or busy image with lots of elements",
|
| 89 |
+
],
|
| 90 |
+
"has_small_details": [
|
| 91 |
+
"an image with fine details, small text, or intricate patterns",
|
| 92 |
+
"a close-up showing texture, small objects, or fine features",
|
| 93 |
+
],
|
| 94 |
+
}
|
| 95 |
+
|
| 96 |
+
def __init__(
|
| 97 |
+
self,
|
| 98 |
+
model_path: str | None = None,
|
| 99 |
+
use_siglip: bool = True,
|
| 100 |
+
device: str | None = None,
|
| 101 |
+
):
|
| 102 |
+
"""Initialize the router.
|
| 103 |
+
|
| 104 |
+
Args:
|
| 105 |
+
model_path: Path to trained model (local or HF hub).
|
| 106 |
+
If None, uses default HF model.
|
| 107 |
+
use_siglip: Whether to use SigLIP for image analysis.
|
| 108 |
+
device: Device to use ('cuda', 'cpu', or None for auto).
|
| 109 |
+
"""
|
| 110 |
+
self.model_path = model_path
|
| 111 |
+
self.use_siglip = use_siglip
|
| 112 |
+
self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
|
| 113 |
+
|
| 114 |
+
# Lazy-loaded models
|
| 115 |
+
self._classifier = None
|
| 116 |
+
self._tokenizer = None
|
| 117 |
+
self._siglip_model = None
|
| 118 |
+
self._siglip_processor = None
|
| 119 |
+
self._text_embeddings = None
|
| 120 |
+
|
| 121 |
+
def is_available(self) -> bool:
|
| 122 |
+
"""Check if required models can be loaded."""
|
| 123 |
+
try:
|
| 124 |
+
self._load_models()
|
| 125 |
+
return True
|
| 126 |
+
except Exception:
|
| 127 |
+
return False
|
| 128 |
+
|
| 129 |
+
def _load_models(self) -> None:
|
| 130 |
+
"""Lazy load the classifier and optionally SigLIP."""
|
| 131 |
+
if self._classifier is None:
|
| 132 |
+
# Determine model path
|
| 133 |
+
if self.model_path:
|
| 134 |
+
model_id = self.model_path
|
| 135 |
+
else:
|
| 136 |
+
# Check for local model first (development)
|
| 137 |
+
local_path = (
|
| 138 |
+
Path(__file__).parent.parent.parent
|
| 139 |
+
/ "models"
|
| 140 |
+
/ "technique-router-mini"
|
| 141 |
+
/ "final"
|
| 142 |
+
)
|
| 143 |
+
if local_path.exists():
|
| 144 |
+
model_id = str(local_path)
|
| 145 |
+
else:
|
| 146 |
+
model_id = self.DEFAULT_HF_MODEL
|
| 147 |
+
|
| 148 |
+
# Load classifier
|
| 149 |
+
self._tokenizer = AutoTokenizer.from_pretrained(model_id)
|
| 150 |
+
self._classifier = AutoModelForSequenceClassification.from_pretrained(model_id)
|
| 151 |
+
self._classifier.to(self.device) # type: ignore[attr-defined]
|
| 152 |
+
self._classifier.eval() # type: ignore[attr-defined]
|
| 153 |
+
|
| 154 |
+
if self.use_siglip and self._siglip_model is None:
|
| 155 |
+
self._siglip_model = AutoModel.from_pretrained(self.SIGLIP_MODEL)
|
| 156 |
+
self._siglip_processor = AutoProcessor.from_pretrained(self.SIGLIP_MODEL)
|
| 157 |
+
self._siglip_model.to(self.device) # type: ignore[attr-defined]
|
| 158 |
+
self._siglip_model.eval() # type: ignore[attr-defined]
|
| 159 |
+
|
| 160 |
+
# Pre-compute text embeddings for image analysis
|
| 161 |
+
self._compute_text_embeddings()
|
| 162 |
+
|
| 163 |
+
def _compute_text_embeddings(self) -> None:
|
| 164 |
+
"""Pre-compute SigLIP text embeddings for image analysis."""
|
| 165 |
+
assert self._siglip_processor is not None
|
| 166 |
+
assert self._siglip_model is not None
|
| 167 |
+
|
| 168 |
+
self._text_embeddings = {}
|
| 169 |
+
|
| 170 |
+
with torch.no_grad():
|
| 171 |
+
for signal_name, descriptions in self.IMAGE_DESCRIPTIONS.items():
|
| 172 |
+
embeddings = []
|
| 173 |
+
for desc in descriptions:
|
| 174 |
+
inputs = self._siglip_processor(
|
| 175 |
+
text=[desc],
|
| 176 |
+
return_tensors="pt",
|
| 177 |
+
padding=True,
|
| 178 |
+
)
|
| 179 |
+
inputs = {k: v.to(self.device) for k, v in inputs.items()}
|
| 180 |
+
text_embeds = self._siglip_model.get_text_features(**inputs)
|
| 181 |
+
text_embeds = text_embeds / text_embeds.norm(dim=-1, keepdim=True)
|
| 182 |
+
embeddings.append(text_embeds)
|
| 183 |
+
|
| 184 |
+
self._text_embeddings[signal_name] = torch.cat(embeddings, dim=0)
|
| 185 |
+
|
| 186 |
+
def _classify_query(self, query: str) -> tuple[Technique, float]:
|
| 187 |
+
"""Classify query intent using trained model.
|
| 188 |
+
|
| 189 |
+
Returns:
|
| 190 |
+
Tuple of (predicted_technique, confidence)
|
| 191 |
+
"""
|
| 192 |
+
self._load_models()
|
| 193 |
+
assert self._tokenizer is not None
|
| 194 |
+
assert self._classifier is not None
|
| 195 |
+
|
| 196 |
+
inputs = self._tokenizer(
|
| 197 |
+
query,
|
| 198 |
+
return_tensors="pt",
|
| 199 |
+
truncation=True,
|
| 200 |
+
padding=True,
|
| 201 |
+
max_length=64,
|
| 202 |
+
)
|
| 203 |
+
inputs = {k: v.to(self.device) for k, v in inputs.items()}
|
| 204 |
+
|
| 205 |
+
with torch.no_grad():
|
| 206 |
+
outputs = self._classifier(**inputs)
|
| 207 |
+
probs = torch.softmax(outputs.logits, dim=-1)
|
| 208 |
+
pred_id = torch.argmax(probs, dim=-1).item()
|
| 209 |
+
confidence = probs[0][pred_id].item()
|
| 210 |
+
|
| 211 |
+
# Map ID to technique
|
| 212 |
+
id2label = self._classifier.config.id2label
|
| 213 |
+
technique_name = id2label[pred_id]
|
| 214 |
+
technique = Technique(technique_name)
|
| 215 |
+
|
| 216 |
+
return technique, confidence
|
| 217 |
+
|
| 218 |
+
def _get_image_embedding(self, image_data: bytes) -> torch.Tensor:
|
| 219 |
+
"""Get SigLIP embedding for image."""
|
| 220 |
+
assert self._siglip_processor is not None
|
| 221 |
+
assert self._siglip_model is not None
|
| 222 |
+
|
| 223 |
+
image = Image.open(io.BytesIO(image_data)).convert("RGB")
|
| 224 |
+
|
| 225 |
+
inputs = self._siglip_processor(
|
| 226 |
+
images=image,
|
| 227 |
+
return_tensors="pt",
|
| 228 |
+
)
|
| 229 |
+
inputs = {k: v.to(self.device) for k, v in inputs.items()}
|
| 230 |
+
|
| 231 |
+
with torch.no_grad():
|
| 232 |
+
image_embeds = self._siglip_model.get_image_features(**inputs)
|
| 233 |
+
image_embeds = image_embeds / image_embeds.norm(dim=-1, keepdim=True)
|
| 234 |
+
|
| 235 |
+
return image_embeds
|
| 236 |
+
|
| 237 |
+
def _analyze_image(self, image_embedding: torch.Tensor) -> ImageSignals:
|
| 238 |
+
"""Analyze image properties using SigLIP."""
|
| 239 |
+
assert self._text_embeddings is not None
|
| 240 |
+
|
| 241 |
+
scores: dict[str, float] = {}
|
| 242 |
+
|
| 243 |
+
def sigmoid(x: float) -> float:
|
| 244 |
+
import math
|
| 245 |
+
|
| 246 |
+
return 1 / (1 + math.exp(-x * 5))
|
| 247 |
+
|
| 248 |
+
with torch.no_grad():
|
| 249 |
+
for signal_name, text_embeds in self._text_embeddings.items():
|
| 250 |
+
# Compute similarity with each description
|
| 251 |
+
similarities = (image_embedding @ text_embeds.T).squeeze(0)
|
| 252 |
+
# Take max similarity across descriptions
|
| 253 |
+
max_sim = similarities.max().item()
|
| 254 |
+
scores[signal_name] = max_sim
|
| 255 |
+
|
| 256 |
+
return ImageSignals(
|
| 257 |
+
has_text=sigmoid(scores["has_text"]),
|
| 258 |
+
is_document=sigmoid(scores["is_document"]),
|
| 259 |
+
is_complex=sigmoid(scores["is_complex"]),
|
| 260 |
+
has_small_details=sigmoid(scores["has_small_details"]),
|
| 261 |
+
)
|
| 262 |
+
|
| 263 |
+
def classify(self, image_data: bytes, query: str) -> RouteDecision:
|
| 264 |
+
"""Classify query + image to determine optimal technique.
|
| 265 |
+
|
| 266 |
+
Args:
|
| 267 |
+
image_data: Raw image bytes
|
| 268 |
+
query: User's query about the image
|
| 269 |
+
|
| 270 |
+
Returns:
|
| 271 |
+
RouteDecision with technique, confidence, and reasoning
|
| 272 |
+
"""
|
| 273 |
+
self._load_models()
|
| 274 |
+
|
| 275 |
+
# Step 1: Classify query with trained model
|
| 276 |
+
technique, query_confidence = self._classify_query(query)
|
| 277 |
+
|
| 278 |
+
# Step 2: Analyze image with SigLIP (if enabled)
|
| 279 |
+
image_signals = None
|
| 280 |
+
if self.use_siglip:
|
| 281 |
+
image_embedding = self._get_image_embedding(image_data)
|
| 282 |
+
image_signals = self._analyze_image(image_embedding)
|
| 283 |
+
|
| 284 |
+
# Step 3: Combine signals for final decision
|
| 285 |
+
final_technique = technique
|
| 286 |
+
confidence = query_confidence
|
| 287 |
+
reason = f"Query classified as '{technique.value}' with {query_confidence:.0%} confidence"
|
| 288 |
+
|
| 289 |
+
# Apply image-based adjustments
|
| 290 |
+
if image_signals:
|
| 291 |
+
# If query says TRANSCODE but image has no text, might want to reconsider
|
| 292 |
+
if technique == Technique.TRANSCODE:
|
| 293 |
+
if image_signals.has_text < 0.4 and image_signals.is_document < 0.4:
|
| 294 |
+
# Low text signal - reduce confidence but keep technique
|
| 295 |
+
# (user explicitly asked for text, they may know better)
|
| 296 |
+
confidence *= 0.8
|
| 297 |
+
reason += " (note: low text detected in image)"
|
| 298 |
+
|
| 299 |
+
# If query says FULL_LOW but image has small details, might need PRESERVE
|
| 300 |
+
elif technique == Technique.FULL_LOW:
|
| 301 |
+
if image_signals.has_small_details > 0.7:
|
| 302 |
+
# Image has fine details - suggest they might need PRESERVE
|
| 303 |
+
reason += " (note: image has fine details, consider PRESERVE)"
|
| 304 |
+
|
| 305 |
+
# If query says PRESERVE, boost confidence if image confirms
|
| 306 |
+
elif technique == Technique.PRESERVE:
|
| 307 |
+
if image_signals.has_small_details > 0.5 or image_signals.is_complex > 0.5:
|
| 308 |
+
confidence = min(1.0, confidence * 1.1)
|
| 309 |
+
reason += " (confirmed: image has fine details)"
|
| 310 |
+
|
| 311 |
+
return RouteDecision(
|
| 312 |
+
technique=final_technique,
|
| 313 |
+
confidence=confidence,
|
| 314 |
+
reason=reason,
|
| 315 |
+
image_signals=image_signals,
|
| 316 |
+
query_prediction=technique.value,
|
| 317 |
+
query_confidence=query_confidence,
|
| 318 |
+
)
|
| 319 |
+
|
| 320 |
+
|
| 321 |
+
def get_trained_router(model_path: str | None = None) -> TrainedRouter:
|
| 322 |
+
"""Get a trained router instance.
|
| 323 |
+
|
| 324 |
+
Args:
|
| 325 |
+
model_path: Optional path to model (local or HF hub).
|
| 326 |
+
If None, uses local model if available, else HF hub.
|
| 327 |
+
"""
|
| 328 |
+
return TrainedRouter(model_path=model_path)
|
headroom/proxy/server.py
CHANGED
|
@@ -87,6 +87,22 @@ from headroom.transforms import (
|
|
| 87 |
is_tree_sitter_available,
|
| 88 |
)
|
| 89 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
# Conditionally import LLMLingua if available
|
| 91 |
if _LLMLINGUA_AVAILABLE:
|
| 92 |
from headroom.transforms import LLMLinguaCompressor, LLMLinguaConfig
|
|
@@ -180,6 +196,7 @@ class ProxyConfig:
|
|
| 180 |
|
| 181 |
# Optimization
|
| 182 |
optimize: bool = True
|
|
|
|
| 183 |
min_tokens_to_crush: int = 500
|
| 184 |
max_items_after_crush: int = 50
|
| 185 |
keep_last_turns: int = 4
|
|
@@ -1198,6 +1215,19 @@ class HeadroomProxy:
|
|
| 1198 |
messages = body.get("messages", [])
|
| 1199 |
stream = body.get("stream", False)
|
| 1200 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1201 |
# Extract headers and tags
|
| 1202 |
headers = dict(request.headers.items())
|
| 1203 |
headers.pop("host", None)
|
|
@@ -2732,6 +2762,19 @@ class HeadroomProxy:
|
|
| 2732 |
messages = body.get("messages", [])
|
| 2733 |
stream = body.get("stream", False)
|
| 2734 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2735 |
headers = dict(request.headers.items())
|
| 2736 |
headers.pop("host", None)
|
| 2737 |
headers.pop("content-length", None)
|
|
|
|
| 87 |
is_tree_sitter_available,
|
| 88 |
)
|
| 89 |
|
| 90 |
+
# Image compression (lazy-loaded to avoid heavy dependencies at startup)
|
| 91 |
+
_image_compressor = None
|
| 92 |
+
|
| 93 |
+
def _get_image_compressor():
|
| 94 |
+
"""Lazy load image compressor to avoid startup overhead."""
|
| 95 |
+
global _image_compressor
|
| 96 |
+
if _image_compressor is None:
|
| 97 |
+
try:
|
| 98 |
+
from headroom.image import ImageCompressor
|
| 99 |
+
_image_compressor = ImageCompressor()
|
| 100 |
+
logger.info("Image compression enabled (model: chopratejas/technique-router)")
|
| 101 |
+
except ImportError as e:
|
| 102 |
+
logger.warning(f"Image compression not available: {e}")
|
| 103 |
+
_image_compressor = False # Mark as unavailable
|
| 104 |
+
return _image_compressor if _image_compressor else None
|
| 105 |
+
|
| 106 |
# Conditionally import LLMLingua if available
|
| 107 |
if _LLMLINGUA_AVAILABLE:
|
| 108 |
from headroom.transforms import LLMLinguaCompressor, LLMLinguaConfig
|
|
|
|
| 196 |
|
| 197 |
# Optimization
|
| 198 |
optimize: bool = True
|
| 199 |
+
image_optimize: bool = True # Compress images using trained ML router
|
| 200 |
min_tokens_to_crush: int = 500
|
| 201 |
max_items_after_crush: int = 50
|
| 202 |
keep_last_turns: int = 4
|
|
|
|
| 1215 |
messages = body.get("messages", [])
|
| 1216 |
stream = body.get("stream", False)
|
| 1217 |
|
| 1218 |
+
# Image compression (before text optimization)
|
| 1219 |
+
if self.config.image_optimize and messages:
|
| 1220 |
+
compressor = _get_image_compressor()
|
| 1221 |
+
if compressor and compressor.has_images(messages):
|
| 1222 |
+
messages = compressor.compress(messages, provider="anthropic")
|
| 1223 |
+
if compressor.last_result:
|
| 1224 |
+
logger.info(
|
| 1225 |
+
f"Image compression: {compressor.last_result.technique.value} "
|
| 1226 |
+
f"({compressor.last_result.savings_percent:.0f}% saved, "
|
| 1227 |
+
f"{compressor.last_result.original_tokens} -> "
|
| 1228 |
+
f"{compressor.last_result.compressed_tokens} tokens)"
|
| 1229 |
+
)
|
| 1230 |
+
|
| 1231 |
# Extract headers and tags
|
| 1232 |
headers = dict(request.headers.items())
|
| 1233 |
headers.pop("host", None)
|
|
|
|
| 2762 |
messages = body.get("messages", [])
|
| 2763 |
stream = body.get("stream", False)
|
| 2764 |
|
| 2765 |
+
# Image compression (before text optimization)
|
| 2766 |
+
if self.config.image_optimize and messages:
|
| 2767 |
+
compressor = _get_image_compressor()
|
| 2768 |
+
if compressor and compressor.has_images(messages):
|
| 2769 |
+
messages = compressor.compress(messages, provider="openai")
|
| 2770 |
+
if compressor.last_result:
|
| 2771 |
+
logger.info(
|
| 2772 |
+
f"Image compression: {compressor.last_result.technique.value} "
|
| 2773 |
+
f"({compressor.last_result.savings_percent:.0f}% saved, "
|
| 2774 |
+
f"{compressor.last_result.original_tokens} -> "
|
| 2775 |
+
f"{compressor.last_result.compressed_tokens} tokens)"
|
| 2776 |
+
)
|
| 2777 |
+
|
| 2778 |
headers = dict(request.headers.items())
|
| 2779 |
headers.pop("host", None)
|
| 2780 |
headers.pop("content-length", None)
|
headroom/transforms/content_router.py
CHANGED
|
@@ -168,6 +168,7 @@ class ContentRouterConfig:
|
|
| 168 |
enable_smart_crusher: Enable JSON array compression.
|
| 169 |
enable_search_compressor: Enable search result compression.
|
| 170 |
enable_log_compressor: Enable build/test log compression.
|
|
|
|
| 171 |
prefer_code_aware_for_code: Use CodeAware over LLMLingua for code.
|
| 172 |
mixed_content_threshold: Min distinct types to consider "mixed".
|
| 173 |
min_section_tokens: Minimum tokens for a section to compress.
|
|
@@ -183,6 +184,7 @@ class ContentRouterConfig:
|
|
| 183 |
enable_smart_crusher: bool = True
|
| 184 |
enable_search_compressor: bool = True
|
| 185 |
enable_log_compressor: bool = True
|
|
|
|
| 186 |
|
| 187 |
# Routing preferences
|
| 188 |
prefer_code_aware_for_code: bool = True
|
|
@@ -413,6 +415,7 @@ class ContentRouter(Transform):
|
|
| 413 |
self._log_compressor: Any = None
|
| 414 |
self._llmlingua: Any = None
|
| 415 |
self._text_compressor: Any = None
|
|
|
|
| 416 |
|
| 417 |
def compress(
|
| 418 |
self,
|
|
@@ -780,6 +783,77 @@ class ContentRouter(Transform):
|
|
| 780 |
logger.debug("TextCompressor not available")
|
| 781 |
return self._text_compressor
|
| 782 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 783 |
# Transform interface
|
| 784 |
|
| 785 |
def apply(
|
|
|
|
| 168 |
enable_smart_crusher: Enable JSON array compression.
|
| 169 |
enable_search_compressor: Enable search result compression.
|
| 170 |
enable_log_compressor: Enable build/test log compression.
|
| 171 |
+
enable_image_optimizer: Enable image token optimization.
|
| 172 |
prefer_code_aware_for_code: Use CodeAware over LLMLingua for code.
|
| 173 |
mixed_content_threshold: Min distinct types to consider "mixed".
|
| 174 |
min_section_tokens: Minimum tokens for a section to compress.
|
|
|
|
| 184 |
enable_smart_crusher: bool = True
|
| 185 |
enable_search_compressor: bool = True
|
| 186 |
enable_log_compressor: bool = True
|
| 187 |
+
enable_image_optimizer: bool = True # Image token optimization
|
| 188 |
|
| 189 |
# Routing preferences
|
| 190 |
prefer_code_aware_for_code: bool = True
|
|
|
|
| 415 |
self._log_compressor: Any = None
|
| 416 |
self._llmlingua: Any = None
|
| 417 |
self._text_compressor: Any = None
|
| 418 |
+
self._image_optimizer: Any = None
|
| 419 |
|
| 420 |
def compress(
|
| 421 |
self,
|
|
|
|
| 783 |
logger.debug("TextCompressor not available")
|
| 784 |
return self._text_compressor
|
| 785 |
|
| 786 |
+
def _get_image_optimizer(self) -> Any:
|
| 787 |
+
"""Get ImageCompressor (lazy load).
|
| 788 |
+
|
| 789 |
+
The ImageCompressor handles image token compression using:
|
| 790 |
+
- Trained MiniLM classifier from HuggingFace (chopratejas/technique-router)
|
| 791 |
+
- SigLIP for image analysis
|
| 792 |
+
- Provider-specific compression (OpenAI detail, Anthropic/Google resize)
|
| 793 |
+
"""
|
| 794 |
+
if self._image_optimizer is None:
|
| 795 |
+
try:
|
| 796 |
+
from ..image import ImageCompressor
|
| 797 |
+
|
| 798 |
+
self._image_optimizer = ImageCompressor()
|
| 799 |
+
except ImportError:
|
| 800 |
+
logger.debug("ImageCompressor not available")
|
| 801 |
+
return self._image_optimizer
|
| 802 |
+
|
| 803 |
+
def optimize_images_in_messages(
|
| 804 |
+
self,
|
| 805 |
+
messages: list[dict[str, Any]],
|
| 806 |
+
tokenizer: Tokenizer,
|
| 807 |
+
provider: str = "openai",
|
| 808 |
+
user_query: str | None = None,
|
| 809 |
+
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
|
| 810 |
+
"""Optimize images in messages.
|
| 811 |
+
|
| 812 |
+
This is a convenience method for image optimization that can be called
|
| 813 |
+
directly or as part of the transform pipeline.
|
| 814 |
+
|
| 815 |
+
Uses ImageCompressor with trained MiniLM router from HuggingFace
|
| 816 |
+
(chopratejas/technique-router) + SigLIP for image analysis.
|
| 817 |
+
|
| 818 |
+
Args:
|
| 819 |
+
messages: Messages potentially containing images.
|
| 820 |
+
tokenizer: Tokenizer for token counting (unused, kept for API compat).
|
| 821 |
+
provider: LLM provider (openai, anthropic, google).
|
| 822 |
+
user_query: User query for task intent detection (unused, auto-extracted).
|
| 823 |
+
|
| 824 |
+
Returns:
|
| 825 |
+
Tuple of (optimized_messages, metrics).
|
| 826 |
+
"""
|
| 827 |
+
if not self.config.enable_image_optimizer:
|
| 828 |
+
return messages, {"images_optimized": 0, "tokens_saved": 0}
|
| 829 |
+
|
| 830 |
+
compressor = self._get_image_optimizer()
|
| 831 |
+
if compressor is None:
|
| 832 |
+
return messages, {"images_optimized": 0, "tokens_saved": 0}
|
| 833 |
+
|
| 834 |
+
# Check if there are images to compress
|
| 835 |
+
if not compressor.has_images(messages):
|
| 836 |
+
return messages, {"images_optimized": 0, "tokens_saved": 0}
|
| 837 |
+
|
| 838 |
+
# Compress images (query is auto-extracted from messages)
|
| 839 |
+
optimized = compressor.compress(messages, provider=provider)
|
| 840 |
+
|
| 841 |
+
# Get metrics from last compression
|
| 842 |
+
result = compressor.last_result
|
| 843 |
+
if result:
|
| 844 |
+
metrics = {
|
| 845 |
+
"images_optimized": result.compressed_tokens < result.original_tokens,
|
| 846 |
+
"tokens_before": result.original_tokens,
|
| 847 |
+
"tokens_after": result.compressed_tokens,
|
| 848 |
+
"tokens_saved": result.original_tokens - result.compressed_tokens,
|
| 849 |
+
"technique": result.technique.value,
|
| 850 |
+
"confidence": result.confidence,
|
| 851 |
+
}
|
| 852 |
+
else:
|
| 853 |
+
metrics = {"images_optimized": 0, "tokens_saved": 0}
|
| 854 |
+
|
| 855 |
+
return optimized, metrics
|
| 856 |
+
|
| 857 |
# Transform interface
|
| 858 |
|
| 859 |
def apply(
|
pyproject.toml
CHANGED
|
@@ -49,6 +49,11 @@ dependencies = [
|
|
| 49 |
"openai>=2.14.0",
|
| 50 |
"sentence-transformers>=5.2.0",
|
| 51 |
"litellm>=1.0.0",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
]
|
| 53 |
|
| 54 |
[project.optional-dependencies]
|
|
|
|
| 49 |
"openai>=2.14.0",
|
| 50 |
"sentence-transformers>=5.2.0",
|
| 51 |
"litellm>=1.0.0",
|
| 52 |
+
"accelerate>=1.12.0",
|
| 53 |
+
"sentencepiece>=0.2.1",
|
| 54 |
+
"protobuf>=6.33.4",
|
| 55 |
+
"semantic-router>=0.1.12",
|
| 56 |
+
"datasets>=4.5.0",
|
| 57 |
]
|
| 58 |
|
| 59 |
[project.optional-dependencies]
|
tests/test_image_compressor.py
ADDED
|
@@ -0,0 +1,927 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Comprehensive tests for the image compression feature.
|
| 2 |
+
|
| 3 |
+
Tests ImageCompressor class and TrainedRouter for:
|
| 4 |
+
- Image detection in various provider formats
|
| 5 |
+
- Query extraction
|
| 6 |
+
- Compression routing
|
| 7 |
+
- Provider-specific compression
|
| 8 |
+
- Edge cases
|
| 9 |
+
- Token estimation
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
import os
|
| 13 |
+
|
| 14 |
+
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
| 15 |
+
|
| 16 |
+
import base64
|
| 17 |
+
import io
|
| 18 |
+
from dataclasses import dataclass
|
| 19 |
+
from enum import Enum
|
| 20 |
+
from typing import Optional
|
| 21 |
+
from unittest.mock import MagicMock, patch, PropertyMock
|
| 22 |
+
|
| 23 |
+
import pytest
|
| 24 |
+
|
| 25 |
+
# Import from PIL for creating test images
|
| 26 |
+
try:
|
| 27 |
+
from PIL import Image
|
| 28 |
+
HAS_PIL = True
|
| 29 |
+
except ImportError:
|
| 30 |
+
HAS_PIL = False
|
| 31 |
+
|
| 32 |
+
from headroom.image.compressor import (
|
| 33 |
+
ImageCompressor,
|
| 34 |
+
Technique,
|
| 35 |
+
CompressionResult,
|
| 36 |
+
compress_images,
|
| 37 |
+
get_compressor,
|
| 38 |
+
)
|
| 39 |
+
from headroom.image.trained_router import (
|
| 40 |
+
TrainedRouter,
|
| 41 |
+
Technique as RouterTechnique,
|
| 42 |
+
RouteDecision,
|
| 43 |
+
ImageSignals,
|
| 44 |
+
)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
# ============================================================================
|
| 48 |
+
# Fixtures
|
| 49 |
+
# ============================================================================
|
| 50 |
+
|
| 51 |
+
@pytest.fixture
|
| 52 |
+
def small_test_image_bytes():
|
| 53 |
+
"""Create a small test image as bytes."""
|
| 54 |
+
if not HAS_PIL:
|
| 55 |
+
pytest.skip("PIL not available")
|
| 56 |
+
|
| 57 |
+
# Create a simple 100x100 red image
|
| 58 |
+
img = Image.new("RGB", (100, 100), color="red")
|
| 59 |
+
buffer = io.BytesIO()
|
| 60 |
+
img.save(buffer, format="PNG")
|
| 61 |
+
return buffer.getvalue()
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
@pytest.fixture
|
| 65 |
+
def large_test_image_bytes():
|
| 66 |
+
"""Create a larger test image as bytes (1024x1024)."""
|
| 67 |
+
if not HAS_PIL:
|
| 68 |
+
pytest.skip("PIL not available")
|
| 69 |
+
|
| 70 |
+
# Create a 1024x1024 image with some pattern
|
| 71 |
+
img = Image.new("RGB", (1024, 1024), color="blue")
|
| 72 |
+
buffer = io.BytesIO()
|
| 73 |
+
img.save(buffer, format="PNG")
|
| 74 |
+
return buffer.getvalue()
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
@pytest.fixture
|
| 78 |
+
def small_image_base64(small_test_image_bytes):
|
| 79 |
+
"""Base64 encoded small test image."""
|
| 80 |
+
return base64.b64encode(small_test_image_bytes).decode("utf-8")
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
@pytest.fixture
|
| 84 |
+
def large_image_base64(large_test_image_bytes):
|
| 85 |
+
"""Base64 encoded large test image."""
|
| 86 |
+
return base64.b64encode(large_test_image_bytes).decode("utf-8")
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
@pytest.fixture
|
| 90 |
+
def openai_messages_with_image(small_image_base64):
|
| 91 |
+
"""Sample OpenAI format messages with image."""
|
| 92 |
+
return [
|
| 93 |
+
{
|
| 94 |
+
"role": "user",
|
| 95 |
+
"content": [
|
| 96 |
+
{"type": "text", "text": "What is in this image?"},
|
| 97 |
+
{
|
| 98 |
+
"type": "image_url",
|
| 99 |
+
"image_url": {
|
| 100 |
+
"url": f"data:image/png;base64,{small_image_base64}",
|
| 101 |
+
"detail": "auto"
|
| 102 |
+
}
|
| 103 |
+
}
|
| 104 |
+
]
|
| 105 |
+
}
|
| 106 |
+
]
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
@pytest.fixture
|
| 110 |
+
def anthropic_messages_with_image(small_image_base64):
|
| 111 |
+
"""Sample Anthropic format messages with image."""
|
| 112 |
+
return [
|
| 113 |
+
{
|
| 114 |
+
"role": "user",
|
| 115 |
+
"content": [
|
| 116 |
+
{"type": "text", "text": "Describe this image"},
|
| 117 |
+
{
|
| 118 |
+
"type": "image",
|
| 119 |
+
"source": {
|
| 120 |
+
"type": "base64",
|
| 121 |
+
"media_type": "image/png",
|
| 122 |
+
"data": small_image_base64
|
| 123 |
+
}
|
| 124 |
+
}
|
| 125 |
+
]
|
| 126 |
+
}
|
| 127 |
+
]
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
@pytest.fixture
|
| 131 |
+
def google_messages_with_image(small_image_base64):
|
| 132 |
+
"""Sample Google format messages with image."""
|
| 133 |
+
return [
|
| 134 |
+
{
|
| 135 |
+
"role": "user",
|
| 136 |
+
"content": [
|
| 137 |
+
{"text": "What do you see?"},
|
| 138 |
+
{
|
| 139 |
+
"inlineData": {
|
| 140 |
+
"mimeType": "image/png",
|
| 141 |
+
"data": small_image_base64
|
| 142 |
+
}
|
| 143 |
+
}
|
| 144 |
+
]
|
| 145 |
+
}
|
| 146 |
+
]
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
@pytest.fixture
|
| 150 |
+
def text_only_messages():
|
| 151 |
+
"""Messages without any images."""
|
| 152 |
+
return [
|
| 153 |
+
{"role": "system", "content": "You are a helpful assistant."},
|
| 154 |
+
{"role": "user", "content": "Hello, how are you?"},
|
| 155 |
+
{"role": "assistant", "content": "I'm doing well, thank you!"},
|
| 156 |
+
{"role": "user", "content": "What is the capital of France?"}
|
| 157 |
+
]
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
@pytest.fixture
|
| 161 |
+
def compressor():
|
| 162 |
+
"""Get an ImageCompressor instance."""
|
| 163 |
+
return ImageCompressor()
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
@pytest.fixture
|
| 167 |
+
def mock_route_decision_full_low():
|
| 168 |
+
"""Mock RouteDecision for FULL_LOW."""
|
| 169 |
+
return RouteDecision(
|
| 170 |
+
technique=RouterTechnique.FULL_LOW,
|
| 171 |
+
confidence=0.9,
|
| 172 |
+
reason="General query about image contents",
|
| 173 |
+
image_signals=None,
|
| 174 |
+
query_prediction="full_low",
|
| 175 |
+
query_confidence=0.9,
|
| 176 |
+
)
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
@pytest.fixture
|
| 180 |
+
def mock_route_decision_preserve():
|
| 181 |
+
"""Mock RouteDecision for PRESERVE."""
|
| 182 |
+
return RouteDecision(
|
| 183 |
+
technique=RouterTechnique.PRESERVE,
|
| 184 |
+
confidence=0.95,
|
| 185 |
+
reason="Query requires fine detail analysis",
|
| 186 |
+
image_signals=None,
|
| 187 |
+
query_prediction="preserve",
|
| 188 |
+
query_confidence=0.95,
|
| 189 |
+
)
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
@pytest.fixture
|
| 193 |
+
def mock_route_decision_transcode():
|
| 194 |
+
"""Mock RouteDecision for TRANSCODE."""
|
| 195 |
+
return RouteDecision(
|
| 196 |
+
technique=RouterTechnique.TRANSCODE,
|
| 197 |
+
confidence=0.88,
|
| 198 |
+
reason="Query asks to read text from image",
|
| 199 |
+
image_signals=None,
|
| 200 |
+
query_prediction="transcode",
|
| 201 |
+
query_confidence=0.88,
|
| 202 |
+
)
|
| 203 |
+
|
| 204 |
+
|
| 205 |
+
@pytest.fixture
|
| 206 |
+
def mock_route_decision_crop():
|
| 207 |
+
"""Mock RouteDecision for CROP."""
|
| 208 |
+
return RouteDecision(
|
| 209 |
+
technique=RouterTechnique.CROP,
|
| 210 |
+
confidence=0.85,
|
| 211 |
+
reason="Query asks about specific region",
|
| 212 |
+
image_signals=None,
|
| 213 |
+
query_prediction="crop",
|
| 214 |
+
query_confidence=0.85,
|
| 215 |
+
)
|
| 216 |
+
|
| 217 |
+
|
| 218 |
+
def create_mock_router(route_decision):
|
| 219 |
+
"""Create a mock router that returns the given decision."""
|
| 220 |
+
mock_router = MagicMock()
|
| 221 |
+
mock_router.classify.return_value = route_decision
|
| 222 |
+
return mock_router
|
| 223 |
+
|
| 224 |
+
|
| 225 |
+
# ============================================================================
|
| 226 |
+
# Test ImageCompressor class - Image detection
|
| 227 |
+
# ============================================================================
|
| 228 |
+
|
| 229 |
+
class TestImageDetection:
|
| 230 |
+
"""Tests for image detection in various formats."""
|
| 231 |
+
|
| 232 |
+
def test_has_images_openai_format(self, compressor, openai_messages_with_image):
|
| 233 |
+
"""Detect images in OpenAI format."""
|
| 234 |
+
assert compressor.has_images(openai_messages_with_image) is True
|
| 235 |
+
|
| 236 |
+
def test_has_images_anthropic_format(self, compressor, anthropic_messages_with_image):
|
| 237 |
+
"""Detect images in Anthropic format."""
|
| 238 |
+
assert compressor.has_images(anthropic_messages_with_image) is True
|
| 239 |
+
|
| 240 |
+
def test_has_images_google_format(self, compressor, google_messages_with_image):
|
| 241 |
+
"""Detect images in Google format."""
|
| 242 |
+
assert compressor.has_images(google_messages_with_image) is True
|
| 243 |
+
|
| 244 |
+
def test_has_images_no_images(self, compressor, text_only_messages):
|
| 245 |
+
"""Returns False when no images in messages."""
|
| 246 |
+
assert compressor.has_images(text_only_messages) is False
|
| 247 |
+
|
| 248 |
+
def test_has_images_empty_messages(self, compressor):
|
| 249 |
+
"""Handles empty message list."""
|
| 250 |
+
assert compressor.has_images([]) is False
|
| 251 |
+
|
| 252 |
+
def test_has_images_string_content(self, compressor):
|
| 253 |
+
"""Handles messages with plain string content."""
|
| 254 |
+
messages = [
|
| 255 |
+
{"role": "user", "content": "Just text, no images"}
|
| 256 |
+
]
|
| 257 |
+
assert compressor.has_images(messages) is False
|
| 258 |
+
|
| 259 |
+
def test_has_images_mixed_content(self, compressor, small_image_base64):
|
| 260 |
+
"""Detect images in messages with mixed content."""
|
| 261 |
+
messages = [
|
| 262 |
+
{"role": "system", "content": "You are helpful."},
|
| 263 |
+
{"role": "user", "content": "What is 2+2?"},
|
| 264 |
+
{
|
| 265 |
+
"role": "user",
|
| 266 |
+
"content": [
|
| 267 |
+
{"type": "text", "text": "Now look at this"},
|
| 268 |
+
{
|
| 269 |
+
"type": "image_url",
|
| 270 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 271 |
+
}
|
| 272 |
+
]
|
| 273 |
+
}
|
| 274 |
+
]
|
| 275 |
+
assert compressor.has_images(messages) is True
|
| 276 |
+
|
| 277 |
+
|
| 278 |
+
# ============================================================================
|
| 279 |
+
# Test ImageCompressor class - Query extraction
|
| 280 |
+
# ============================================================================
|
| 281 |
+
|
| 282 |
+
class TestQueryExtraction:
|
| 283 |
+
"""Tests for extracting text query from messages."""
|
| 284 |
+
|
| 285 |
+
def test_extract_query_from_openai_format(self, compressor, openai_messages_with_image):
|
| 286 |
+
"""Extracts text query from OpenAI format messages."""
|
| 287 |
+
query = compressor._extract_query(openai_messages_with_image)
|
| 288 |
+
assert query == "What is in this image?"
|
| 289 |
+
|
| 290 |
+
def test_extract_query_from_anthropic_format(self, compressor, anthropic_messages_with_image):
|
| 291 |
+
"""Extracts text query from Anthropic format messages."""
|
| 292 |
+
query = compressor._extract_query(anthropic_messages_with_image)
|
| 293 |
+
assert query == "Describe this image"
|
| 294 |
+
|
| 295 |
+
def test_extract_query_empty_string_when_no_text(self, compressor, small_image_base64):
|
| 296 |
+
"""Returns empty string when no text in user message."""
|
| 297 |
+
messages = [
|
| 298 |
+
{
|
| 299 |
+
"role": "user",
|
| 300 |
+
"content": [
|
| 301 |
+
{
|
| 302 |
+
"type": "image_url",
|
| 303 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 304 |
+
}
|
| 305 |
+
]
|
| 306 |
+
}
|
| 307 |
+
]
|
| 308 |
+
query = compressor._extract_query(messages)
|
| 309 |
+
assert query == ""
|
| 310 |
+
|
| 311 |
+
def test_extract_query_from_plain_text_message(self, compressor):
|
| 312 |
+
"""Extracts query from plain text user message."""
|
| 313 |
+
messages = [
|
| 314 |
+
{"role": "user", "content": "What is this?"}
|
| 315 |
+
]
|
| 316 |
+
query = compressor._extract_query(messages)
|
| 317 |
+
assert query == "What is this?"
|
| 318 |
+
|
| 319 |
+
def test_extract_query_uses_last_user_message(self, compressor):
|
| 320 |
+
"""Extracts query from the most recent user message."""
|
| 321 |
+
messages = [
|
| 322 |
+
{"role": "user", "content": "First question"},
|
| 323 |
+
{"role": "assistant", "content": "First answer"},
|
| 324 |
+
{"role": "user", "content": "Second question"}
|
| 325 |
+
]
|
| 326 |
+
query = compressor._extract_query(messages)
|
| 327 |
+
assert query == "Second question"
|
| 328 |
+
|
| 329 |
+
|
| 330 |
+
# ============================================================================
|
| 331 |
+
# Test ImageCompressor class - Image data extraction
|
| 332 |
+
# ============================================================================
|
| 333 |
+
|
| 334 |
+
class TestImageDataExtraction:
|
| 335 |
+
"""Tests for extracting base64 image data from messages."""
|
| 336 |
+
|
| 337 |
+
def test_extract_image_data_openai_format(self, compressor, openai_messages_with_image, small_test_image_bytes):
|
| 338 |
+
"""Extracts base64 image data from OpenAI format."""
|
| 339 |
+
data = compressor._extract_image_data(openai_messages_with_image)
|
| 340 |
+
assert data is not None
|
| 341 |
+
assert isinstance(data, bytes)
|
| 342 |
+
# Verify it's valid image data
|
| 343 |
+
assert data == small_test_image_bytes
|
| 344 |
+
|
| 345 |
+
def test_extract_image_data_anthropic_format(self, compressor, anthropic_messages_with_image, small_test_image_bytes):
|
| 346 |
+
"""Extracts base64 image data from Anthropic format."""
|
| 347 |
+
data = compressor._extract_image_data(anthropic_messages_with_image)
|
| 348 |
+
assert data is not None
|
| 349 |
+
assert isinstance(data, bytes)
|
| 350 |
+
assert data == small_test_image_bytes
|
| 351 |
+
|
| 352 |
+
def test_extract_image_data_google_format(self, compressor, google_messages_with_image, small_test_image_bytes):
|
| 353 |
+
"""Extracts base64 image data from Google format."""
|
| 354 |
+
data = compressor._extract_image_data(google_messages_with_image)
|
| 355 |
+
assert data is not None
|
| 356 |
+
assert isinstance(data, bytes)
|
| 357 |
+
assert data == small_test_image_bytes
|
| 358 |
+
|
| 359 |
+
def test_extract_image_data_returns_none_for_text_only(self, compressor, text_only_messages):
|
| 360 |
+
"""Returns None when no images in messages."""
|
| 361 |
+
data = compressor._extract_image_data(text_only_messages)
|
| 362 |
+
assert data is None
|
| 363 |
+
|
| 364 |
+
def test_extract_image_data_returns_first_image(self, compressor, small_image_base64):
|
| 365 |
+
"""Extracts the first image when multiple images present."""
|
| 366 |
+
messages = [
|
| 367 |
+
{
|
| 368 |
+
"role": "user",
|
| 369 |
+
"content": [
|
| 370 |
+
{
|
| 371 |
+
"type": "image_url",
|
| 372 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 373 |
+
},
|
| 374 |
+
{
|
| 375 |
+
"type": "image_url",
|
| 376 |
+
"image_url": {"url": "data:image/png;base64,SECOND_IMAGE_DATA"}
|
| 377 |
+
}
|
| 378 |
+
]
|
| 379 |
+
}
|
| 380 |
+
]
|
| 381 |
+
data = compressor._extract_image_data(messages)
|
| 382 |
+
assert data is not None
|
| 383 |
+
|
| 384 |
+
|
| 385 |
+
# ============================================================================
|
| 386 |
+
# Test Compression routing
|
| 387 |
+
# ============================================================================
|
| 388 |
+
|
| 389 |
+
class TestCompressionRouting:
|
| 390 |
+
"""Tests for compression technique routing based on query."""
|
| 391 |
+
|
| 392 |
+
def test_compress_general_query(self, compressor, openai_messages_with_image, mock_route_decision_full_low):
|
| 393 |
+
"""'What is this?' query routes to full_low technique."""
|
| 394 |
+
mock_router = create_mock_router(mock_route_decision_full_low)
|
| 395 |
+
|
| 396 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 397 |
+
result = compressor.compress(openai_messages_with_image, "openai")
|
| 398 |
+
|
| 399 |
+
# Verify the router was called
|
| 400 |
+
mock_router.classify.assert_called_once()
|
| 401 |
+
|
| 402 |
+
# For FULL_LOW, OpenAI should get detail="low"
|
| 403 |
+
content = result[0]["content"]
|
| 404 |
+
for item in content:
|
| 405 |
+
if item.get("type") == "image_url":
|
| 406 |
+
assert item["image_url"].get("detail") == "low"
|
| 407 |
+
|
| 408 |
+
def test_compress_detail_query(self, compressor, small_image_base64, mock_route_decision_preserve):
|
| 409 |
+
"""'Count the whiskers' query routes to preserve technique."""
|
| 410 |
+
messages = [
|
| 411 |
+
{
|
| 412 |
+
"role": "user",
|
| 413 |
+
"content": [
|
| 414 |
+
{"type": "text", "text": "Count the whiskers on the cat"},
|
| 415 |
+
{
|
| 416 |
+
"type": "image_url",
|
| 417 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 418 |
+
}
|
| 419 |
+
]
|
| 420 |
+
}
|
| 421 |
+
]
|
| 422 |
+
mock_router = create_mock_router(mock_route_decision_preserve)
|
| 423 |
+
|
| 424 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 425 |
+
result = compressor.compress(messages, "openai")
|
| 426 |
+
mock_router.classify.assert_called_once()
|
| 427 |
+
|
| 428 |
+
def test_compress_text_query(self, compressor, small_image_base64, mock_route_decision_transcode):
|
| 429 |
+
"""'Read the text' query routes to transcode technique."""
|
| 430 |
+
messages = [
|
| 431 |
+
{
|
| 432 |
+
"role": "user",
|
| 433 |
+
"content": [
|
| 434 |
+
{"type": "text", "text": "Read the text in this document"},
|
| 435 |
+
{
|
| 436 |
+
"type": "image_url",
|
| 437 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 438 |
+
}
|
| 439 |
+
]
|
| 440 |
+
}
|
| 441 |
+
]
|
| 442 |
+
mock_router = create_mock_router(mock_route_decision_transcode)
|
| 443 |
+
|
| 444 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 445 |
+
result = compressor.compress(messages, "openai")
|
| 446 |
+
mock_router.classify.assert_called_once()
|
| 447 |
+
|
| 448 |
+
def test_compress_region_query(self, compressor, small_image_base64, mock_route_decision_crop):
|
| 449 |
+
"""'What's in the corner?' query routes to crop technique."""
|
| 450 |
+
messages = [
|
| 451 |
+
{
|
| 452 |
+
"role": "user",
|
| 453 |
+
"content": [
|
| 454 |
+
{"type": "text", "text": "What's in the top-left corner?"},
|
| 455 |
+
{
|
| 456 |
+
"type": "image_url",
|
| 457 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 458 |
+
}
|
| 459 |
+
]
|
| 460 |
+
}
|
| 461 |
+
]
|
| 462 |
+
mock_router = create_mock_router(mock_route_decision_crop)
|
| 463 |
+
|
| 464 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 465 |
+
result = compressor.compress(messages, "openai")
|
| 466 |
+
mock_router.classify.assert_called_once()
|
| 467 |
+
|
| 468 |
+
|
| 469 |
+
# ============================================================================
|
| 470 |
+
# Test Provider-specific compression
|
| 471 |
+
# ============================================================================
|
| 472 |
+
|
| 473 |
+
class TestProviderSpecificCompression:
|
| 474 |
+
"""Tests for provider-specific image compression."""
|
| 475 |
+
|
| 476 |
+
def test_openai_detail_low(self, compressor, openai_messages_with_image, mock_route_decision_full_low):
|
| 477 |
+
"""OpenAI: sets detail='low' for full_low technique."""
|
| 478 |
+
mock_router = create_mock_router(mock_route_decision_full_low)
|
| 479 |
+
|
| 480 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 481 |
+
result = compressor.compress(openai_messages_with_image, "openai")
|
| 482 |
+
|
| 483 |
+
# Find the image item and check detail
|
| 484 |
+
for item in result[0]["content"]:
|
| 485 |
+
if item.get("type") == "image_url":
|
| 486 |
+
assert item["image_url"]["detail"] == "low"
|
| 487 |
+
|
| 488 |
+
def test_openai_detail_preserved(self, compressor, small_image_base64, mock_route_decision_preserve):
|
| 489 |
+
"""OpenAI: preserves original detail setting for preserve technique."""
|
| 490 |
+
messages = [
|
| 491 |
+
{
|
| 492 |
+
"role": "user",
|
| 493 |
+
"content": [
|
| 494 |
+
{"type": "text", "text": "Analyze fine details"},
|
| 495 |
+
{
|
| 496 |
+
"type": "image_url",
|
| 497 |
+
"image_url": {
|
| 498 |
+
"url": f"data:image/png;base64,{small_image_base64}",
|
| 499 |
+
"detail": "high"
|
| 500 |
+
}
|
| 501 |
+
}
|
| 502 |
+
]
|
| 503 |
+
}
|
| 504 |
+
]
|
| 505 |
+
mock_router = create_mock_router(mock_route_decision_preserve)
|
| 506 |
+
|
| 507 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 508 |
+
result = compressor.compress(messages, "openai")
|
| 509 |
+
|
| 510 |
+
# For preserve, the image should remain unchanged
|
| 511 |
+
for item in result[0]["content"]:
|
| 512 |
+
if item.get("type") == "image_url":
|
| 513 |
+
# Should keep original high detail
|
| 514 |
+
detail = item["image_url"].get("detail")
|
| 515 |
+
assert detail == "high"
|
| 516 |
+
|
| 517 |
+
def test_anthropic_format(self, compressor, anthropic_messages_with_image, mock_route_decision_full_low):
|
| 518 |
+
"""Handles Anthropic image format correctly."""
|
| 519 |
+
mock_router = create_mock_router(mock_route_decision_full_low)
|
| 520 |
+
|
| 521 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 522 |
+
result = compressor.compress(anthropic_messages_with_image, "anthropic")
|
| 523 |
+
|
| 524 |
+
# Should return valid messages (may or may not transform Anthropic format)
|
| 525 |
+
assert isinstance(result, list)
|
| 526 |
+
assert len(result) > 0
|
| 527 |
+
|
| 528 |
+
|
| 529 |
+
# ============================================================================
|
| 530 |
+
# Test Edge cases
|
| 531 |
+
# ============================================================================
|
| 532 |
+
|
| 533 |
+
class TestEdgeCases:
|
| 534 |
+
"""Tests for edge cases and error handling."""
|
| 535 |
+
|
| 536 |
+
def test_no_images_passthrough(self, compressor, text_only_messages):
|
| 537 |
+
"""Returns messages unchanged if no images present."""
|
| 538 |
+
result = compressor.compress(text_only_messages, "openai")
|
| 539 |
+
assert result == text_only_messages
|
| 540 |
+
|
| 541 |
+
def test_empty_messages(self, compressor):
|
| 542 |
+
"""Handles empty message list gracefully."""
|
| 543 |
+
result = compressor.compress([], "openai")
|
| 544 |
+
assert result == []
|
| 545 |
+
|
| 546 |
+
def test_router_failure_fallback(self, compressor, openai_messages_with_image):
|
| 547 |
+
"""Falls back to preserve technique on router error."""
|
| 548 |
+
mock_router = MagicMock()
|
| 549 |
+
mock_router.classify.side_effect = Exception("Router failed")
|
| 550 |
+
|
| 551 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 552 |
+
# Should not raise, should fall back gracefully
|
| 553 |
+
result = compressor.compress(openai_messages_with_image, "openai")
|
| 554 |
+
|
| 555 |
+
# Messages should be returned (either original or with preserve)
|
| 556 |
+
assert isinstance(result, list)
|
| 557 |
+
assert len(result) > 0
|
| 558 |
+
|
| 559 |
+
def test_invalid_base64_data(self, compressor, mock_route_decision_preserve):
|
| 560 |
+
"""Handles invalid base64 data gracefully."""
|
| 561 |
+
messages = [
|
| 562 |
+
{
|
| 563 |
+
"role": "user",
|
| 564 |
+
"content": [
|
| 565 |
+
{"type": "text", "text": "What is this?"},
|
| 566 |
+
{
|
| 567 |
+
"type": "image_url",
|
| 568 |
+
"image_url": {"url": "data:image/png;base64,bm90X3ZhbGlkX2ltYWdlX2RhdGE="}
|
| 569 |
+
}
|
| 570 |
+
]
|
| 571 |
+
}
|
| 572 |
+
]
|
| 573 |
+
|
| 574 |
+
# Use a mock router to avoid actual model loading
|
| 575 |
+
mock_router = create_mock_router(mock_route_decision_preserve)
|
| 576 |
+
|
| 577 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 578 |
+
# Should not raise
|
| 579 |
+
result = compressor.compress(messages, "openai")
|
| 580 |
+
assert isinstance(result, list)
|
| 581 |
+
|
| 582 |
+
def test_url_image_not_base64(self, compressor):
|
| 583 |
+
"""Handles URL-based images (not base64)."""
|
| 584 |
+
messages = [
|
| 585 |
+
{
|
| 586 |
+
"role": "user",
|
| 587 |
+
"content": [
|
| 588 |
+
{"type": "text", "text": "What is this?"},
|
| 589 |
+
{
|
| 590 |
+
"type": "image_url",
|
| 591 |
+
"image_url": {"url": "https://example.com/image.jpg"}
|
| 592 |
+
}
|
| 593 |
+
]
|
| 594 |
+
}
|
| 595 |
+
]
|
| 596 |
+
|
| 597 |
+
# URL images should just pass through since we can't extract data
|
| 598 |
+
result = compressor.compress(messages, "openai")
|
| 599 |
+
assert isinstance(result, list)
|
| 600 |
+
# Should return original messages since no base64 data to extract
|
| 601 |
+
assert result == messages
|
| 602 |
+
|
| 603 |
+
def test_none_content(self, compressor):
|
| 604 |
+
"""Handles messages with None content."""
|
| 605 |
+
messages = [
|
| 606 |
+
{"role": "user", "content": None}
|
| 607 |
+
]
|
| 608 |
+
|
| 609 |
+
result = compressor.compress(messages, "openai")
|
| 610 |
+
assert result == messages
|
| 611 |
+
|
| 612 |
+
def test_missing_content_key(self, compressor):
|
| 613 |
+
"""Handles messages missing content key."""
|
| 614 |
+
messages = [
|
| 615 |
+
{"role": "user"}
|
| 616 |
+
]
|
| 617 |
+
|
| 618 |
+
result = compressor.compress(messages, "openai")
|
| 619 |
+
assert result == messages
|
| 620 |
+
|
| 621 |
+
|
| 622 |
+
# ============================================================================
|
| 623 |
+
# Test Token estimation
|
| 624 |
+
# ============================================================================
|
| 625 |
+
|
| 626 |
+
class TestTokenEstimation:
|
| 627 |
+
"""Tests for image token estimation."""
|
| 628 |
+
|
| 629 |
+
def test_estimate_tokens_small_image(self, compressor, small_test_image_bytes):
|
| 630 |
+
"""Estimates tokens for a small image correctly."""
|
| 631 |
+
# Pass actual image bytes, not base64
|
| 632 |
+
# 100x100 image with low detail = 85 tokens
|
| 633 |
+
tokens = compressor._estimate_tokens(small_test_image_bytes, "low")
|
| 634 |
+
assert tokens == 85
|
| 635 |
+
|
| 636 |
+
def test_estimate_tokens_large_image(self, compressor, large_test_image_bytes):
|
| 637 |
+
"""Estimates tokens for a large image correctly."""
|
| 638 |
+
# 1024x1024 image with high detail
|
| 639 |
+
# tiles_x = ceil(1024/512) = 2
|
| 640 |
+
# tiles_y = ceil(1024/512) = 2
|
| 641 |
+
# tokens = 85 * 2 * 2 + 170 = 510
|
| 642 |
+
tokens = compressor._estimate_tokens(large_test_image_bytes, "high")
|
| 643 |
+
assert tokens == 510
|
| 644 |
+
|
| 645 |
+
def test_estimate_tokens_low_detail_constant(self, compressor, large_test_image_bytes):
|
| 646 |
+
"""Low detail always returns 85 tokens regardless of size."""
|
| 647 |
+
tokens = compressor._estimate_tokens(large_test_image_bytes, "low")
|
| 648 |
+
assert tokens == 85
|
| 649 |
+
|
| 650 |
+
def test_savings_calculation(self):
|
| 651 |
+
"""CompressionResult calculates savings percentage correctly."""
|
| 652 |
+
result = CompressionResult(
|
| 653 |
+
technique=Technique.FULL_LOW,
|
| 654 |
+
original_tokens=1000,
|
| 655 |
+
compressed_tokens=85,
|
| 656 |
+
confidence=0.9
|
| 657 |
+
)
|
| 658 |
+
|
| 659 |
+
# (1000 - 85) / 1000 * 100 = 91.5%
|
| 660 |
+
assert result.savings_percent == pytest.approx(91.5, rel=0.01)
|
| 661 |
+
|
| 662 |
+
def test_savings_zero_original_tokens(self):
|
| 663 |
+
"""Handles zero original tokens without division error."""
|
| 664 |
+
result = CompressionResult(
|
| 665 |
+
technique=Technique.PRESERVE,
|
| 666 |
+
original_tokens=0,
|
| 667 |
+
compressed_tokens=0,
|
| 668 |
+
confidence=1.0
|
| 669 |
+
)
|
| 670 |
+
|
| 671 |
+
assert result.savings_percent == 0.0
|
| 672 |
+
|
| 673 |
+
def test_estimate_tokens_invalid_data(self, compressor):
|
| 674 |
+
"""Returns default token count for invalid image data."""
|
| 675 |
+
# Pass invalid bytes that can't be opened as image
|
| 676 |
+
tokens = compressor._estimate_tokens(b"invalid_image_data", "high")
|
| 677 |
+
# Should return a default value (765 based on the code)
|
| 678 |
+
assert tokens == 765
|
| 679 |
+
|
| 680 |
+
|
| 681 |
+
# ============================================================================
|
| 682 |
+
# Test TrainedRouter (mocked)
|
| 683 |
+
# ============================================================================
|
| 684 |
+
|
| 685 |
+
class TestTrainedRouterMocked:
|
| 686 |
+
"""Tests for TrainedRouter with mocked model loading."""
|
| 687 |
+
|
| 688 |
+
def test_router_technique_enum_values(self):
|
| 689 |
+
"""Verify Technique enum has expected values."""
|
| 690 |
+
assert RouterTechnique.FULL_LOW.value == "full_low"
|
| 691 |
+
assert RouterTechnique.PRESERVE.value == "preserve"
|
| 692 |
+
assert RouterTechnique.TRANSCODE.value == "transcode"
|
| 693 |
+
assert RouterTechnique.CROP.value == "crop"
|
| 694 |
+
|
| 695 |
+
def test_route_decision_dataclass(self):
|
| 696 |
+
"""Verify RouteDecision dataclass structure."""
|
| 697 |
+
decision = RouteDecision(
|
| 698 |
+
technique=RouterTechnique.FULL_LOW,
|
| 699 |
+
confidence=0.9,
|
| 700 |
+
reason="Test reason",
|
| 701 |
+
image_signals=None,
|
| 702 |
+
query_prediction="full_low",
|
| 703 |
+
query_confidence=0.9
|
| 704 |
+
)
|
| 705 |
+
|
| 706 |
+
assert decision.technique == RouterTechnique.FULL_LOW
|
| 707 |
+
assert decision.confidence == 0.9
|
| 708 |
+
assert decision.reason == "Test reason"
|
| 709 |
+
|
| 710 |
+
def test_image_signals_dataclass(self):
|
| 711 |
+
"""Verify ImageSignals dataclass structure."""
|
| 712 |
+
signals = ImageSignals(
|
| 713 |
+
has_text=0.8,
|
| 714 |
+
is_document=0.6,
|
| 715 |
+
is_complex=0.3,
|
| 716 |
+
has_small_details=0.2
|
| 717 |
+
)
|
| 718 |
+
|
| 719 |
+
assert signals.has_text == 0.8
|
| 720 |
+
assert signals.is_document == 0.6
|
| 721 |
+
assert signals.is_complex == 0.3
|
| 722 |
+
assert signals.has_small_details == 0.2
|
| 723 |
+
|
| 724 |
+
@patch("headroom.image.trained_router.AutoModelForSequenceClassification")
|
| 725 |
+
@patch("headroom.image.trained_router.AutoTokenizer")
|
| 726 |
+
def test_router_is_available_with_models(self, mock_tokenizer, mock_model):
|
| 727 |
+
"""Router reports available when models can load."""
|
| 728 |
+
mock_tokenizer.from_pretrained.return_value = MagicMock()
|
| 729 |
+
mock_model.from_pretrained.return_value = MagicMock()
|
| 730 |
+
|
| 731 |
+
router = TrainedRouter()
|
| 732 |
+
|
| 733 |
+
# Mock _load_models to not actually load
|
| 734 |
+
with patch.object(router, '_load_models'):
|
| 735 |
+
assert router.is_available() is True
|
| 736 |
+
|
| 737 |
+
def test_router_is_available_false_on_error(self):
|
| 738 |
+
"""Router reports not available when models fail to load."""
|
| 739 |
+
router = TrainedRouter(model_path="/nonexistent/path")
|
| 740 |
+
|
| 741 |
+
# This should return False since the model path doesn't exist
|
| 742 |
+
# and loading will fail
|
| 743 |
+
with patch.object(router, '_load_models', side_effect=Exception("Model not found")):
|
| 744 |
+
assert router.is_available() is False
|
| 745 |
+
|
| 746 |
+
|
| 747 |
+
# ============================================================================
|
| 748 |
+
# Test Convenience functions
|
| 749 |
+
# ============================================================================
|
| 750 |
+
|
| 751 |
+
class TestConvenienceFunctions:
|
| 752 |
+
"""Tests for module-level convenience functions."""
|
| 753 |
+
|
| 754 |
+
def test_get_compressor_returns_instance(self):
|
| 755 |
+
"""get_compressor returns an ImageCompressor instance."""
|
| 756 |
+
compressor = get_compressor()
|
| 757 |
+
assert isinstance(compressor, ImageCompressor)
|
| 758 |
+
|
| 759 |
+
def test_get_compressor_singleton(self):
|
| 760 |
+
"""get_compressor returns the same instance."""
|
| 761 |
+
compressor1 = get_compressor()
|
| 762 |
+
compressor2 = get_compressor()
|
| 763 |
+
assert compressor1 is compressor2
|
| 764 |
+
|
| 765 |
+
def test_compress_images_function(self, text_only_messages):
|
| 766 |
+
"""compress_images convenience function works."""
|
| 767 |
+
result = compress_images(text_only_messages, "openai")
|
| 768 |
+
assert result == text_only_messages
|
| 769 |
+
|
| 770 |
+
|
| 771 |
+
# ============================================================================
|
| 772 |
+
# Integration tests (with mocked router)
|
| 773 |
+
# ============================================================================
|
| 774 |
+
|
| 775 |
+
class TestIntegration:
|
| 776 |
+
"""Integration tests with mocked router."""
|
| 777 |
+
|
| 778 |
+
def test_full_compression_flow_openai(self, small_image_base64, mock_route_decision_full_low):
|
| 779 |
+
"""Test complete compression flow for OpenAI format."""
|
| 780 |
+
messages = [
|
| 781 |
+
{
|
| 782 |
+
"role": "user",
|
| 783 |
+
"content": [
|
| 784 |
+
{"type": "text", "text": "What is this?"},
|
| 785 |
+
{
|
| 786 |
+
"type": "image_url",
|
| 787 |
+
"image_url": {
|
| 788 |
+
"url": f"data:image/png;base64,{small_image_base64}",
|
| 789 |
+
"detail": "auto"
|
| 790 |
+
}
|
| 791 |
+
}
|
| 792 |
+
]
|
| 793 |
+
}
|
| 794 |
+
]
|
| 795 |
+
|
| 796 |
+
compressor = ImageCompressor()
|
| 797 |
+
mock_router = create_mock_router(mock_route_decision_full_low)
|
| 798 |
+
|
| 799 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 800 |
+
result = compressor.compress(messages, "openai")
|
| 801 |
+
|
| 802 |
+
# Verify structure
|
| 803 |
+
assert len(result) == 1
|
| 804 |
+
assert result[0]["role"] == "user"
|
| 805 |
+
assert isinstance(result[0]["content"], list)
|
| 806 |
+
|
| 807 |
+
# Verify image was processed
|
| 808 |
+
has_image = False
|
| 809 |
+
for item in result[0]["content"]:
|
| 810 |
+
if item.get("type") == "image_url":
|
| 811 |
+
has_image = True
|
| 812 |
+
assert item["image_url"]["detail"] == "low"
|
| 813 |
+
assert has_image
|
| 814 |
+
|
| 815 |
+
def test_full_compression_flow_anthropic(self, small_image_base64, mock_route_decision_full_low):
|
| 816 |
+
"""Test complete compression flow for Anthropic format."""
|
| 817 |
+
messages = [
|
| 818 |
+
{
|
| 819 |
+
"role": "user",
|
| 820 |
+
"content": [
|
| 821 |
+
{"type": "text", "text": "Describe this"},
|
| 822 |
+
{
|
| 823 |
+
"type": "image",
|
| 824 |
+
"source": {
|
| 825 |
+
"type": "base64",
|
| 826 |
+
"media_type": "image/png",
|
| 827 |
+
"data": small_image_base64
|
| 828 |
+
}
|
| 829 |
+
}
|
| 830 |
+
]
|
| 831 |
+
}
|
| 832 |
+
]
|
| 833 |
+
|
| 834 |
+
compressor = ImageCompressor()
|
| 835 |
+
mock_router = create_mock_router(mock_route_decision_full_low)
|
| 836 |
+
|
| 837 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 838 |
+
result = compressor.compress(messages, "anthropic")
|
| 839 |
+
|
| 840 |
+
# Should return valid messages
|
| 841 |
+
assert len(result) == 1
|
| 842 |
+
assert result[0]["role"] == "user"
|
| 843 |
+
|
| 844 |
+
def test_multiple_images_in_message(self, small_image_base64, mock_route_decision_full_low):
|
| 845 |
+
"""Test compression with multiple images."""
|
| 846 |
+
messages = [
|
| 847 |
+
{
|
| 848 |
+
"role": "user",
|
| 849 |
+
"content": [
|
| 850 |
+
{"type": "text", "text": "Compare these images"},
|
| 851 |
+
{
|
| 852 |
+
"type": "image_url",
|
| 853 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 854 |
+
},
|
| 855 |
+
{
|
| 856 |
+
"type": "image_url",
|
| 857 |
+
"image_url": {"url": f"data:image/png;base64,{small_image_base64}"}
|
| 858 |
+
}
|
| 859 |
+
]
|
| 860 |
+
}
|
| 861 |
+
]
|
| 862 |
+
|
| 863 |
+
compressor = ImageCompressor()
|
| 864 |
+
mock_router = create_mock_router(mock_route_decision_full_low)
|
| 865 |
+
|
| 866 |
+
with patch.object(compressor, '_get_router', return_value=mock_router):
|
| 867 |
+
result = compressor.compress(messages, "openai")
|
| 868 |
+
|
| 869 |
+
# Both images should be processed
|
| 870 |
+
image_count = 0
|
| 871 |
+
for item in result[0]["content"]:
|
| 872 |
+
if item.get("type") == "image_url":
|
| 873 |
+
image_count += 1
|
| 874 |
+
assert item["image_url"]["detail"] == "low"
|
| 875 |
+
assert image_count == 2
|
| 876 |
+
|
| 877 |
+
|
| 878 |
+
# ============================================================================
|
| 879 |
+
# ContentRouter Integration Tests
|
| 880 |
+
# ============================================================================
|
| 881 |
+
|
| 882 |
+
|
| 883 |
+
class TestContentRouterIntegration:
|
| 884 |
+
"""Test ImageCompressor integration with ContentRouter."""
|
| 885 |
+
|
| 886 |
+
def test_content_router_loads_image_compressor(self):
|
| 887 |
+
"""Verify ContentRouter can load ImageCompressor (not None)."""
|
| 888 |
+
from headroom.transforms.content_router import ContentRouter
|
| 889 |
+
|
| 890 |
+
router = ContentRouter()
|
| 891 |
+
compressor = router._get_image_optimizer()
|
| 892 |
+
|
| 893 |
+
# This should NOT be None - if it is, the import failed silently
|
| 894 |
+
assert compressor is not None, (
|
| 895 |
+
"ContentRouter._get_image_optimizer() returned None. "
|
| 896 |
+
"This means ImageCompressor import failed silently!"
|
| 897 |
+
)
|
| 898 |
+
|
| 899 |
+
def test_content_router_compressor_is_image_compressor(self):
|
| 900 |
+
"""Verify ContentRouter uses ImageCompressor (not old ImageOptimizer)."""
|
| 901 |
+
from headroom.image import ImageCompressor
|
| 902 |
+
from headroom.transforms.content_router import ContentRouter
|
| 903 |
+
|
| 904 |
+
router = ContentRouter()
|
| 905 |
+
compressor = router._get_image_optimizer()
|
| 906 |
+
|
| 907 |
+
assert isinstance(compressor, ImageCompressor), (
|
| 908 |
+
f"Expected ImageCompressor, got {type(compressor).__name__}"
|
| 909 |
+
)
|
| 910 |
+
|
| 911 |
+
def test_content_router_optimize_images_works(self):
|
| 912 |
+
"""Test optimize_images_in_messages returns valid result."""
|
| 913 |
+
from headroom.transforms.content_router import ContentRouter
|
| 914 |
+
from unittest.mock import MagicMock
|
| 915 |
+
|
| 916 |
+
router = ContentRouter()
|
| 917 |
+
tokenizer = MagicMock()
|
| 918 |
+
|
| 919 |
+
# Simple message without images
|
| 920 |
+
messages = [{"role": "user", "content": "Hello"}]
|
| 921 |
+
result, metrics = router.optimize_images_in_messages(
|
| 922 |
+
messages, tokenizer, provider="openai"
|
| 923 |
+
)
|
| 924 |
+
|
| 925 |
+
assert result == messages
|
| 926 |
+
assert "images_optimized" in metrics
|
| 927 |
+
assert metrics["tokens_saved"] == 0
|
uv.lock
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|