--- license: mit language: - en library_name: sentence-transformers pipeline_tag: sentence-similarity tags: - code - finance datasets: - mlabonne/FineTome-100k - leeroy-jankins/Regulations - leeroy-jankins/Appropriations - leeroy-jankins/OMB-Circular-A-11 - leeroy-jankins/RedBook - leeroy-jankins/SF133 - leeroy-jankins/US-General-Ledger - leeroy-jankins/Title-31-CFR-Money-and-Finance base_model: - ChristianAzinn/mxbai-embed-large-v1-gguf --- Preview - Bobo — Embedding Model (derived from mixedbread-ai/mxbai-embed-large-v1) Bobo is a text-embedding, LLM derived from mixedbread-ai/mxbai-embed-large-v1, packaged for drop-in use in semantic search, RAG (Retrieval-Augmented Generation), clustering, deduplication, and zero-shot classification. It produces dense vector representations suitable for ANN indexes (FAISS, ScaNN, Milvus, pgvector) and common retrieval stacks. ## Key Features - Derived from the proven mixedbread-ai/mxbai-embed-large-v1 embedding family. - Strong performance on retrieval-style tasks (semantic search, RAG, clustering). - Sentence-Transformers–compatible API for fast adoption. - Mean pooling with optional L2 normalization for cosine similarity workflows. - Production-friendly guidance on chunking, batching, and indexing. ## Technical Specifications | Property | Value / Guidance | |--------------------------|-----------------------------------------------------------------------------------| | Base model | mixedbread-ai/mxbai-embed-large-v1 (encoder-style transformer) | | Architecture | Transformer encoder (Sentence-Transformers compatible) | | Embedding dimension | Query programmatically at runtime; common builds use 1024 | | Tokenization | Provided by upstream model tokenizer | | Max input length | Depends on upstream config; chunk long docs (e.g., 256–512 tokens) | | Pooling | Mean pooling (recommended), then optional L2 normalization | | Output | Dense float vectors (often normalized if using cosine similarity) | | Intended backends | FAISS, Milvus, pgvector, Qdrant, Chroma, Weaviate | Tip: Always detect the dimension in code (see Quickstart) and configure your index accordingly. ## 🎯 Quickstart - Here, we provide several ways to produce sentence embeddings. Please note that you have to provide the prompt `Represent this sentence for searching relevant passages: ` for query if you want to use it for retrieval. Besides that you don't need any prompt. ## ⚙️ Fine-Tuneing Datasets | File Name | Description | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------| | [Balanced Budget and Emergency Deficit Control Act of 1985](https://huggingface.co/datasets/leeroy-jankins/The-Balanced-Budget-And-Emergency-Deficit-Control-Act-of-1985) | Establishes statutory limits on federal spending and deficit control mechanisms, including sequestration procedures. | | [Budget Control Act of 2011](https://huggingface.co/datasets/leeroy-jankins/The-Budget-Control-Act-2011) | Sets discretionary spending caps and establishes enforcement mechanisms to control federal deficits. | | [Digital Accountability And Transparency Act of 2014](https://huggingface.co/datasets/leeroy-jankins/Data-Act-2014) | Requires standardized federal spending data and improved transparency through government-wide financial reporting. | | [Federal Account Symbols And Titles Book](https://huggingface.co/datasets/leeroy-jankins/FastBook) | Defines Treasury account symbols and official titles used for federal budgetary and accounting purposes. | | [Federal Acquisition Regulation](https://huggingface.co/datasets/leeroy-jankins/Federal-Acquisition-Regulation) | Establishes uniform policies and procedures governing the acquisition of goods and services by federal agencies. | | [Federal Government Standards For Internal Controls](https://huggingface.co/datasets/leeroy-jankins/Federal-Government-Standards-For-Internal-Controls) | Defines the internal control framework for federal agencies to ensure accountability, integrity, and compliance. | | [Federal Managers Financial Integrity Act of 1982](https://huggingface.co/datasets/leeroy-jankins/FMFIA-1982) | Requires agencies to establish internal controls and report annually on their effectiveness. | | [Federal Trust Fund Accounting Guide](https://huggingface.co/datasets/leeroy-jankins/Federal-Trust-Fund-Accounting-Guide) | Provides accounting guidance for the management and reporting of federal trust funds. | | [Financial Management Regulations DOD 7000-14-R](https://huggingface.co/datasets/leeroy-jankins/DOD-7000-14-Financial-Management-Regulation) | Establishes DoD-specific financial management policies, procedures, and accounting requirements. | | [Fiscal Responsibility Act](https://huggingface.co/datasets/leeroy-jankins/The-Fiscal-Responsibility-Act-of-2023) | Establishes statutory measures intended to improve fiscal discipline and control federal spending. | | [Government Auditing Standards](https://huggingface.co/datasets/leeroy-jankins/Government-Auditing-Standards) | Sets professional standards for audits of government organizations, programs, activities, and functions. | | [Government Invoicing User Guide](https://huggingface.co/datasets/leeroy-jankins/Government-Performance-and-Results-Act) | Provides guidance on federal invoicing standards and processes for government transactions. | | [Government Performance and Results Act of 1993](https://huggingface.co/datasets/leeroy-jankins/Government-Performance-and-Results-Act) | Requires agencies to engage in strategic planning and performance measurement to improve program effectiveness. | | [GPRA Modernization Act of 2010](https://huggingface.co/datasets/leeroy-jankins/The-GPRA-Modernization-Act-Of-2010) | Updates GPRA by strengthening performance management, cross-agency goals, and accountability. | | [OMB Circular A-11 Preparation Submission And Execution Of The Budget](https://huggingface.co/datasets/leeroy-jankins/OMB-Circular-A-11) | Provides comprehensive guidance for preparing, submitting, and executing the President’s Budget. | | [OMB Circular A-11 Section 120 Apportionment Process](https://huggingface.co/datasets/leeroy-jankins/OMB-Circular-A11-Section-120-Apportionment-Process) | Defines the apportionment process used to control the rate of obligation of budgetary resources. | | [OMB Circular A-123 Managements Responsibility for Enterprise Risk Management and Internal Control](https://huggingface.co/datasets/leeroy-jankins/OMB-Circular-A-123) | Defines management responsibilities for internal control and enterprise risk management across federal agencies. | | [Federal Trust Fund Accounting Guide](https://huggingface.co/datasets/leeroy-jankins/Federal-Trust-Fund-Accounting-Guide) | Establishes requirements for federal agency financial statements and reporting. | | [Principles Of Federal Appropriations Law Volume One](https://huggingface.co/datasets/leeroy-jankins/Principles-Of-Federal-Appropriations-Law) | Authoritative GAO guidance on foundational principles governing the use of federal appropriations. | | [Statements of Federal Federal Financial Accounting Concepts and Standards](https://huggingface.co/datasets/leeroy-jankins/Statements-Of-Federal-Financial-Accounting-Concepts-And-Standards) | Establishes accounting concepts and standards for federal financial reporting. | | [The Anti-Deficiency Act PL 97-258](https://huggingface.co/datasets/leeroy-jankins/The-Anti-Deficiency-Act) | Prohibits federal agencies from obligating or expending funds in excess of appropriations or before enactment. | | [The Anti-Deficiency Reform and Enforcement Act of 2018](https://huggingface.co/datasets/leeroy-jankins/The-Anti-Deficiency-Reform-And-Enforcement-Act-Of-2018) | Strengthens Anti-Deficiency Act enforcement and reporting requirements to improve fiscal accountability. | | [The Chief Financial Officers Act of 1990](https://huggingface.co/datasets/leeroy-jankins/The-Chief-Financial-Officers-Act-1990) | Establishes agency Chief Financial Officers and modernizes federal financial management practices. | | [The Congressional Budget and Impoundment Control Act of 1974](https://huggingface.co/datasets/leeroy-jankins/The-Congressional-Budget-And-Impoundment-Control-Act-Of-1974) | Establishes the congressional budget process and restricts executive impoundment of appropriated funds. | | [Statutory Pay As You Go Act of 2010](https://huggingface.co/datasets/leeroy-jankins/Statutory-Pay-As-You-Go-Act-of-2010) | Authorizes interagency agreements for the provision of goods and services on a reimbursable basis. | | [The Stafford Act](https://huggingface.co/datasets/leeroy-jankins/The-Stafford-Act) | Provides the statutory framework for federal disaster response and emergency assistance. | | [Federal Trust Fund Accounting Guide](https://huggingface.co/datasets/leeroy-jankins/Federal-Trust-Fund-Accounting-Guide) | Provides additional appropriations authority beyond regular annual funding acts. | | [Title 2 Code of Federal Regulations – Uniform Administrative Requirements, Cost Principles, and Audit](https://huggingface.co/datasets/leeroy-jankins/Title-2-CFR-Uniform-Administrative-Requirements-Cost-Principles-And-Audit) | Establishes uniform administrative, cost, and audit requirements for federal financial assistance. | | [Title 31 Code of Federal Regulations – Money and Finance](https://huggingface.co/datasets/leeroy-jankins/Title-31-CFR-Money-and-Finance) | Codifies Treasury and federal financial management regulations governing money and finance. | | [US Standard General Ledger Account Definitions](https://huggingface.co/datasets/leeroy-jankins/US-Standard-General-Ledger-Accounts-And-Definitions) | Defines standardized account structures used for federal accounting and financial reporting. | ## 🏗️ Sentence Transformers ``` python -m pip install -U sentence-transformers ``` ```python from sentence_transformers import SentenceTransformer from sentence_transformers.util import cos_sim from sentence_transformers.quantization import quantize_embeddings # 1. Specify preffered dimensions dimensions = 512 # 2. load model model = SentenceTransformer("bobo/mxbai-embed-large-v1", truncate_dim=dimensions) # The prompt used for query retrieval tasks: # query_prompt = 'Represent this sentence for searching relevant passages: ' query = "A man is eating a piece of bread" docs = [ "A man is eating food.", "A man is eating pasta.", "The girl is carrying a baby.", "A man is riding a horse.", ] # 2. Encode query_embedding = model.encode(query, prompt_name="query") # Equivalent Alternatives: # query_embedding = model.encode(query_prompt + query) # query_embedding = model.encode(query, prompt=query_prompt) docs_embeddings = model.encode(docs) # Optional: Quantize the embeddings binary_query_embedding = quantize_embeddings(query_embedding, precision="ubinary") binary_docs_embeddings = quantize_embeddings(docs_embeddings, precision="ubinary") similarities = cos_sim(query_embedding, docs_embeddings) print('similarities:', similarities) ``` ## 🧠 Transformers ```python from typing import Dict import torch import numpy as np from transformers import AutoModel, AutoTokenizer from sentence_transformers.util import cos_sim # For retrieval you need to pass this prompt. Please find our more in our blog post. def transform_query(query: str) -> str: """ For retrieval, add the prompt for query (not for documents). """ return f'Represent this sentence for searching relevant passages: {query}' # The model works really well with cls pooling (default) but also with mean pooling. def pooling(outputs: torch.Tensor, inputs: Dict, strategy: str = 'cls') -> np.ndarray: if strategy == 'cls': outputs = outputs[:, 0] elif strategy == 'mean': outputs = torch.sum( outputs * inputs["attention_mask"][:, :, None], dim=1) / torch.sum(inputs["attention_mask"], dim=1, keepdim=True) else: raise NotImplementedError return outputs.detach().cpu().numpy() # 1. load model model_id = 'bobo/mxbai-embed-large-v1' tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModel.from_pretrained(model_id).cuda() docs = [ transform_query('A man is eating a piece of bread'), "A man is eating food.", "A man is eating pasta.", "The girl is carrying a baby.", "A man is riding a horse.", ] # 2. encode inputs = tokenizer(docs, padding=True, return_tensors='pt') for k, v in inputs.items(): inputs[k] = v.cuda() outputs = model(**inputs).last_hidden_state embeddings = pooling(outputs, inputs, 'cls') similarities = cos_sim(embeddings[0], embeddings[1:]) print('similarities:', similarities) ``` ## 🏁 Evaluation As of March 2024, our model archives SOTA performance for Bert-large sized models on the [MTEB](https://huggingface.co/spaces/mteb/leaderboard). It ourperforms commercial models like OpenAIs text-embedding-3-large and matches the performance of model 20x it's size like the [echo-mistral-7b](https://huggingface.co/jspringer/echo-mistral-7b-instruct-lasttoken). Our model was trained with no overlap of the MTEB data, which indicates that our model generalizes well across several domains, tasks and text length. We know there are some limitations with this model, which will be fixed in v2. | Model | Avg (56 datasets) | Classification (12 datasets) | Clustering (11 datasets) | PairClassification (3 datasets) | Reranking (4 datasets) | Retrieval (15 datasets) | STS (10 datasets) | Summarization (1 dataset) | | --------------------------------------------------------------------------------------------- | ----------------- | ---------------------------- | ------------------------ | ------------------------------- | ---------------------- | ----------------------- | ----------------- | ------------------------- | | **leeroy-jankins/bobo/mxbai-embed-large-v1** | **64.68** | 75.64 | 46.71 | 87.2 | 60.11 | 54.39 | 85.00 | 32.71 | | [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | 64.23 | 75.97 | 46.08 | 87.12 | 60.03 | 54.29 | 83.11 | 31.61 | | [leeroy-jankins/bobo/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/bobo-embed-2d-large-v1) | 63.25 | 74.14 | 46.07 | 85.89 | 58.94 | 51.42 | 84.9 | 31.55 | | [nomic-embed-text-v1](https://huggingface.co/nomic-ai/nomic-embed-text-v1) | 62.39 | 74.12 | 43.91 | 85.15 | 55.69 | 52.81 | 82.06 | 30.08 | | [jina-embeddings-v2-base-en](https://huggingface.co/jinaai/jina-embeddings-v2-base-en) | 60.38 | 73.45 | 41.73 | 85.38 | 56.98 | 47.87 | 80.7 | 31.6 | | *Proprietary Models* | | | | | | | | | | [OpenAI text-embedding-3-large](https://openai.com/blog/new-embedding-models-and-api-updates) | 64.58 | 75.45 | 49.01 | 85.72 | 59.16 | 55.44 | 81.73 | 29.92 | | [Cohere embed-english-v3.0](https://txt.cohere.com/introducing-embed-v3/) | 64.47 | 76.49 | 47.43 | 85.84 | 58.01 | 55.00 | 82.62 | 30.18 | | [OpenAI text-embedding-ada-002](https://openai.com/blog/new-and-improved-embedding-model) | 60.99 | 70.93 | 45.90 | 84.89 | 56.32 | 49.25 | 80.97 | 30.80 | ## 💻 Matryoshka and Binary Quantization Embeddings in their commonly used form (float arrays) have a high memory footprint when used at scale. Two approaches to solve this problem are Matryoshka Representation Learning (MRL) and (Binary) Quantization. While MRL reduces the number of dimensions of an embedding, binary quantization transforms the value of each dimension from a float32 into a lower precision (int8 or even binary). The model supports both approaches! You can also take it one step further, and combine both MRL and quantization. This combination of binary quantization and MRL allows you to reduce the memory usage of your embeddings significantly. This leads to much lower costs when using a vector database in particular. ## 📝License - Bobo is published under the [MIT General Public License v3](https://huggingface.co/leeroy-jankins/leeroy/blob/main/LICENSE.txt)