Commit History
-
Improve retrieval streaming and tracing
fa7c98c
-
fix(retrieval): store the BM25 index as gzipped JSON instead of pickle (#13)
Omar Solano +1: Claude Fable 5Unverified signatureHF's malware scanner flagged the public bundle's pkl as unsafe: BM25 vocabulary tokens from Python docs ("subprocess", "urllib.request", ...) are byte-identical to the STACK_GLOBAL import operands its signatures hunt for. A pickle also asks public downloaders to trust arbitrary code execution on load. The index is now a versioned gzipped-JSON payload (68 MB -> 21 MB) with full parity verified against the old pkl (equal postings, frequencies, lengths; identical search scores). Both HF bundles already ship the .json.gz and the .pkl was removed; stale local bundles self-heal because _bundle_complete() checks the new path. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>e0d6ca8
-
fix(retrieval): budget skip, standard RRF, fail-fast collection load
Three hybrid-retrieval fixes: - _apply_token_budget broke on the first over-budget result, so one oversized rank-1 document (retrieve_doc under a small per-request budget) returned zero context even when smaller lower-ranked chunks fit. Skip the oversized result and keep filling the budget. - reciprocal_rank_fusion accumulated one contribution per occurrence of a dedupe key within a single ranked list, so a section split into several chunks in one retriever's top-k masqueraded as cross-retriever consensus. Standard RRF: each key counts once per list at its best rank; representative selection is unchanged. - LocalChromaRetriever used get_or_create_collection, silently creating an empty collection from a broken or mismatched bundle and degrading dense search to zero hits with no signal. Use get_collection and raise a clear error naming the collection and db path. Also extracts the 0.10 rerank floor as RERANK_SCORE_FLOOR for reuse by the GraphRAG backend. Co-Authore…0f36b6a
-
Add Part C eval variants, LLM judge, and harness fixes
Part C memory/context-management experiment layer: - evals/judge.py: blinded LLM judge (run + validate) that auto-fills the handgrade sheet and gates at >=90% TP/TN vs human labels - app: Part C variants in memory_presets.py + middlewares (sliding_window, prompt_compression, selective_retention, context_reset, observation_truncation, clear_retrieval_kb, incontext_history_retrieval), plus retrieval-budget and disable-kb knobs threaded through chat_service, chat_types, chroma_rag, prompts - telemetry: per-turn signal registry so per-call-view middlewares report compaction in context_stats - evals harness: generalized compaction-signal gate (common.py), md5 sheet_row_id, drop dead prune_incomplete params; tests/test_memory_variants.py - docs: evals_part_c_plan.md; evals.md Part C findings + part_* download fix Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>c344089
-
Enhance knowledge base and API functionality
- Updated .gitignore to include local environment files for better configuration management. - Enhanced the build_chat_request function to handle explicit empty source selections, ensuring that the knowledge base is correctly disabled when no sources are selected. - Improved the UIMessageStreamEncoder to include additional metadata for source matches, enhancing the information available to the frontend. - Introduced reranking logic in the rerank_results function to score matched chunks instead of entire documents, optimizing relevance and performance. - Added comprehensive tests to validate the new functionalities, ensuring robustness in knowledge base interactions and API responses.f2e8e50
-
Enhance error handling in BM25 index loading
- Added exception handling when loading the BM25 index to log warnings for failures and unexpected types. - Implemented fallback to dense-only retrieval if loading fails or if the index type is incorrect, improving robustness.1f970f0 -
Renamed 'scripts' folder to 'app', removed mongo and instructor libraries
bd180ae