Spaces:
Build error
Build error
Commit History
-
Add multi-provider batch API support with CCR post-processing
This commit adds comprehensive batch API support for all three major LLM providers (Anthropic, OpenAI, Google/Gemini) with integrated CCR (Compress-Cache-Retrieve) functionality for asynchronous batch processing. ## Batch CCR Post-Processing Architecture When batch APIs are used, responses are processed asynchronously. If the model calls the CCR retrieval tool (`headroom_retrieve`) within a batch response, the system now handles this automatically: 1. **Batch Submit**: Request context (messages, tools, model) is stored in BatchContextStore keyed by batch_id 2. **Batch Results**: When results are retrieved, CCR tool calls are detected in the responses 3. **Continuation**: For each CCR tool call, the system executes local retrieval and makes a continuation API call to complete the response 4. **Result Update**: The batch result is updated with the complete response, transparent to the caller ## New Components - `headroom/ccr/batch_store.py`: TTL-based context storage for…8756398
