| --- |
| language: |
| - en |
| license: mit |
| tags: |
| - sentence-transformers |
| - sparse-encoder |
| - sparse |
| - splade |
| - generated_from_trainer |
| - dataset_size:800000 |
| - loss:SpladeLoss |
| - loss:SparseMultipleNegativesRankingLoss |
| - loss:FlopsLoss |
| base_model: prajjwal1/bert-tiny |
| widget: |
| - text: how much do private cleaners charge per hour |
| - text: atlantic ocean air currents affects climate |
| - text: RNA polymerase is the core enzyme in transcription which needs proteins known |
| as transcription factors to bind to the DNA promoter. also DNA plymerase... can't |
| remember what about it though. DNA polymerase isn't involved in DNA transcription. |
| However, DNA polymerase IS involved in DNA REPLICATION. |
| - text: Exploit:JS/Axpergle.E Virus is a threatening Trojan horse which gets itself |
| loaded when you turn on your computer and eats up lots of system resources. Once |
| this Exploit:JS/Axpergle.E virus successfully enters your operating system, your |
| computer will be subjected to a variety of errors and drive you mad.) Exploit:JS/Axpergle.E |
| Virus corrupts the data and files saved on your computer hard drive terribly. |
| 2) Exploit:JS/Axpergle.E Virus changes the registry entry to get itself launched |
| at system startup. |
| - text: --No depreciation deduction shall be allowed under this section (and no depreciation |
| or amortization deduction shall be allowed under any other provision of this subtitle) |
| to the taxpayer for any term interest in property for any period during which |
| the remainder interest in such property is held (directly or indirectly) by a |
| related person. |
| pipeline_tag: feature-extraction |
| library_name: sentence-transformers |
| metrics: |
| - dot_accuracy@1 |
| - dot_accuracy@3 |
| - dot_accuracy@5 |
| - dot_accuracy@10 |
| - dot_precision@1 |
| - dot_precision@3 |
| - dot_precision@5 |
| - dot_precision@10 |
| - dot_recall@1 |
| - dot_recall@3 |
| - dot_recall@5 |
| - dot_recall@10 |
| - dot_ndcg@10 |
| - dot_mrr@10 |
| - dot_map@100 |
| - query_active_dims |
| - query_sparsity_ratio |
| - corpus_active_dims |
| - corpus_sparsity_ratio |
| model-index: |
| - name: SPLADE-BERT-Tiny |
| results: |
| - task: |
| type: sparse-information-retrieval |
| name: Sparse Information Retrieval |
| dataset: |
| name: Unknown |
| type: unknown |
| metrics: |
| - type: dot_accuracy@1 |
| value: 0.457 |
| name: Dot Accuracy@1 |
| - type: dot_accuracy@3 |
| value: 0.7572 |
| name: Dot Accuracy@3 |
| - type: dot_accuracy@5 |
| value: 0.8574 |
| name: Dot Accuracy@5 |
| - type: dot_accuracy@10 |
| value: 0.929 |
| name: Dot Accuracy@10 |
| - type: dot_precision@1 |
| value: 0.457 |
| name: Dot Precision@1 |
| - type: dot_precision@3 |
| value: 0.25906666666666667 |
| name: Dot Precision@3 |
| - type: dot_precision@5 |
| value: 0.178 |
| name: Dot Precision@5 |
| - type: dot_precision@10 |
| value: 0.09714 |
| name: Dot Precision@10 |
| - type: dot_recall@1 |
| value: 0.44155 |
| name: Dot Recall@1 |
| - type: dot_recall@3 |
| value: 0.7427833333333334 |
| name: Dot Recall@3 |
| - type: dot_recall@5 |
| value: 0.8471666666666666 |
| name: Dot Recall@5 |
| - type: dot_recall@10 |
| value: 0.9223 |
| name: Dot Recall@10 |
| - type: dot_ndcg@10 |
| value: 0.6931598312411338 |
| name: Dot Ndcg@10 |
| - type: dot_mrr@10 |
| value: 0.6234866666666686 |
| name: Dot Mrr@10 |
| - type: dot_map@100 |
| value: 0.6191148055389254 |
| name: Dot Map@100 |
| - type: query_active_dims |
| value: 21.215999603271484 |
| name: Query Active Dims |
| - type: query_sparsity_ratio |
| value: 0.9993048948429568 |
| name: Query Sparsity Ratio |
| - type: corpus_active_dims |
| value: 159.5419082486014 |
| name: Corpus Active Dims |
| - type: corpus_sparsity_ratio |
| value: 0.99477288813811 |
| name: Corpus Sparsity Ratio |
| --- |
| |
| # SPLADE-BERT-Tiny |
|
|
| This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [prajjwal1/bert-tiny](https://huggingface.co/prajjwal1/bert-tiny) using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval. |
| ## Model Details |
|
|
| ### Model Description |
| - **Model Type:** SPLADE Sparse Encoder |
| - **Base model:** [prajjwal1/bert-tiny](https://huggingface.co/prajjwal1/bert-tiny) <!-- at revision 6f75de8b60a9f8a2fdf7b69cbd86d9e64bcb3837 --> |
| - **Maximum Sequence Length:** 512 tokens |
| - **Output Dimensionality:** 30522 dimensions |
| - **Similarity Function:** Dot Product |
| <!-- - **Training Dataset:** Unknown --> |
| - **Language:** en |
| - **License:** mit |
|
|
| ### Model Sources |
|
|
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
| - **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) |
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) |
| - **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder) |
|
|
| ### Full Model Architecture |
|
|
| ``` |
| SparseEncoder( |
| (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'}) |
| (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) |
| ) |
| ``` |
|
|
| ## Usage |
|
|
| ### Direct Usage (Sentence Transformers) |
|
|
| First install the Sentence Transformers library: |
|
|
| ```bash |
| pip install -U sentence-transformers |
| ``` |
|
|
| Then you can load this model and run inference. |
| ```python |
| from sentence_transformers import SparseEncoder |
| |
| # Download from the 🤗 Hub |
| model = SparseEncoder("yosefw/SPLADE-BERT-Tiny-v2") |
| # Run inference |
| queries = [ |
| "what code section is depreciation", |
| ] |
| documents = [ |
| 'Section 179 depreciation deduction. Section 179 of the United States Internal Revenue Code (26 U.S.C. § 179), allows a taxpayer to elect to deduct the cost of certain types of property on their income taxes as an expense, rather than requiring the cost of the property to be capitalized and depreciated.', |
| '--No depreciation deduction shall be allowed under this section (and no depreciation or amortization deduction shall be allowed under any other provision of this subtitle) to the taxpayer for any term interest in property for any period during which the remainder interest in such property is held (directly or indirectly) by a related person.', |
| 'Depreciation - Amortization Code. Refer to the IRS Instructions for Form 4562, Line 42, for the amortization code.', |
| ] |
| query_embeddings = model.encode_query(queries) |
| document_embeddings = model.encode_document(documents) |
| print(query_embeddings.shape, document_embeddings.shape) |
| # [1, 30522] [3, 30522] |
| |
| # Get the similarity scores for the embeddings |
| similarities = model.similarity(query_embeddings, document_embeddings) |
| print(similarities) |
| # tensor([[17.0167, 11.4943, 13.8083]]) |
| ``` |
|
|
| <!-- |
| ### Direct Usage (Transformers) |
|
|
| <details><summary>Click to see the direct usage in Transformers</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Downstream Usage (Sentence Transformers) |
|
|
| You can finetune this model on your own dataset. |
|
|
| <details><summary>Click to expand</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Out-of-Scope Use |
|
|
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* |
| --> |
|
|
| ## Evaluation |
|
|
| ### Metrics |
|
|
| #### Sparse Information Retrieval |
|
|
| * Evaluated with [<code>SparseInformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator) |
|
|
| | Metric | Value | |
| |:----------------------|:-----------| |
| | dot_accuracy@1 | 0.457 | |
| | dot_accuracy@3 | 0.7572 | |
| | dot_accuracy@5 | 0.8574 | |
| | dot_accuracy@10 | 0.929 | |
| | dot_precision@1 | 0.457 | |
| | dot_precision@3 | 0.2591 | |
| | dot_precision@5 | 0.178 | |
| | dot_precision@10 | 0.0971 | |
| | dot_recall@1 | 0.4415 | |
| | dot_recall@3 | 0.7428 | |
| | dot_recall@5 | 0.8472 | |
| | dot_recall@10 | 0.9223 | |
| | **dot_ndcg@10** | **0.6932** | |
| | dot_mrr@10 | 0.6235 | |
| | dot_map@100 | 0.6191 | |
| | query_active_dims | 21.216 | |
| | query_sparsity_ratio | 0.9993 | |
| | corpus_active_dims | 159.5419 | |
| | corpus_sparsity_ratio | 0.9948 | |
| |
| <!-- |
| ## Bias, Risks and Limitations |
| |
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
| --> |
| |
| <!-- |
| ### Recommendations |
| |
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
| --> |
| |
| ## Training Details |
| |
| ### Training Dataset |
| |
| #### Unnamed Dataset |
| |
| * Size: 800,000 training samples |
| * Columns: <code>query</code>, <code>positive</code>, <code>negative_1</code>, and <code>negative_2</code> |
| * Approximate statistics based on the first 1000 samples: |
| | | query | positive | negative_1 | negative_2 | |
| |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| |
| | type | string | string | string | string | |
| | details | <ul><li>min: 4 tokens</li><li>mean: 9.03 tokens</li><li>max: 30 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 81.92 tokens</li><li>max: 220 tokens</li></ul> | <ul><li>min: 22 tokens</li><li>mean: 78.63 tokens</li><li>max: 227 tokens</li></ul> | <ul><li>min: 18 tokens</li><li>mean: 78.11 tokens</li><li>max: 236 tokens</li></ul> | |
| * Samples: |
| | query | positive | negative_1 | negative_2 | |
| |:-------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | <code>definition of vas deferens</code> | <code>Vas deferens: The tube that connects the testes with the urethra. The vas deferens is a coiled duct that conveys sperm from the epididymis to the ejaculatory duct and the urethra.</code> | <code>For further discussion of the vas deferens within the context of the structures and functions of reproduction and sexuality, please see the overview section “The Reproductive System.”. See also FERTILITY; TESTICLES; VASECTOMY.</code> | <code>1 Testicular cancer symptoms include a painless lump or swelling in a testicle, testicle or scrotum pain, a dull ache in the abdomen, back, or groin, and. 2 Urinary Tract Infections (UTIs) A urinary tract infection (UTI) is an infection of the bladder, kidneys, ureters, or urethra.</code> | |
| | <code>how old is kieron williamson</code> | <code>Kieron Williamson – the latest artist to be part of GoGoDragons! April 21, 2015. A 12-year-old artist, nicknamed Mini-Monet, is to unveil a sculpture of a dragon he has painted for GoGoDragons. Kieron Williamson, from Norfolk, who has so far earned about £2m, painted the 5ft-tall (1.5m) dragon for the event in Norwich.</code> | <code>8-year-old artist: Don't call me Monet. London, England (CNN) -- He has the deft brush strokes of a seasoned artist, but Kieron Williamson is just eight years old. The boy from Norfolk, in eastern England, has been hailed by the British press as a mini Monet, a reference to the famous French impressionist.</code> | <code>Needless to say, this site does not tell you much about his football career (yet!), but the website will tell you everything there is to know about Kieron Williamson’s passion for oil, watercolour and pastel,</code> | |
| | <code>when do you start showing third pregnancy</code> | <code>Yes | No Thank you! I am pregnant with my third child and I am definitly showing at 10 weeks. I am starting to wear some maternity clothes. My low low rise pre-pregnancy jeans still work. My biggest problem is shirts, but fortunately the style right now is loose shirts that look maternity.</code> | <code>Some women do not start to show until they are well into their second trimester or even the start of their third trimester. If you are overweight at the start of your pregnancy, you may not gain as much weight during your pregnancy and may not begin to show until later into your pregnancy. Average: 3.591215.</code> | <code>There isn't a set time when moms-to-be start sporting an obviously pregnant belly – every woman is different. Some women keep their pre-pregnancy belly far into the second trimester, while others start showing in the first trimester.</code> | |
| * Loss: [<code>SpladeLoss</code>](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters: |
| ```json |
| { |
| "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')", |
| "document_regularizer_weight": 0.003, |
| "query_regularizer_weight": 0.005 |
| } |
| ``` |
| |
| ### Training Hyperparameters |
| #### Non-Default Hyperparameters |
| |
| - `eval_strategy`: epoch |
| - `per_device_train_batch_size`: 16 |
| - `per_device_eval_batch_size`: 16 |
| - `gradient_accumulation_steps`: 4 |
| - `learning_rate`: 6e-05 |
| - `num_train_epochs`: 6 |
| - `lr_scheduler_type`: cosine |
| - `warmup_ratio`: 0.025 |
| - `fp16`: True |
| - `load_best_model_at_end`: True |
| - `optim`: adamw_torch_fused |
| - `push_to_hub`: True |
| - `batch_sampler`: no_duplicates |
| |
| #### All Hyperparameters |
| <details><summary>Click to expand</summary> |
| |
| - `overwrite_output_dir`: False |
| - `do_predict`: False |
| - `eval_strategy`: epoch |
| - `prediction_loss_only`: True |
| - `per_device_train_batch_size`: 16 |
| - `per_device_eval_batch_size`: 16 |
| - `per_gpu_train_batch_size`: None |
| - `per_gpu_eval_batch_size`: None |
| - `gradient_accumulation_steps`: 4 |
| - `eval_accumulation_steps`: None |
| - `torch_empty_cache_steps`: None |
| - `learning_rate`: 6e-05 |
| - `weight_decay`: 0.0 |
| - `adam_beta1`: 0.9 |
| - `adam_beta2`: 0.999 |
| - `adam_epsilon`: 1e-08 |
| - `max_grad_norm`: 1.0 |
| - `num_train_epochs`: 6 |
| - `max_steps`: -1 |
| - `lr_scheduler_type`: cosine |
| - `lr_scheduler_kwargs`: {} |
| - `warmup_ratio`: 0.025 |
| - `warmup_steps`: 0 |
| - `log_level`: passive |
| - `log_level_replica`: warning |
| - `log_on_each_node`: True |
| - `logging_nan_inf_filter`: True |
| - `save_safetensors`: True |
| - `save_on_each_node`: False |
| - `save_only_model`: False |
| - `restore_callback_states_from_checkpoint`: False |
| - `no_cuda`: False |
| - `use_cpu`: False |
| - `use_mps_device`: False |
| - `seed`: 42 |
| - `data_seed`: None |
| - `jit_mode_eval`: False |
| - `use_ipex`: False |
| - `bf16`: False |
| - `fp16`: True |
| - `fp16_opt_level`: O1 |
| - `half_precision_backend`: auto |
| - `bf16_full_eval`: False |
| - `fp16_full_eval`: False |
| - `tf32`: None |
| - `local_rank`: 0 |
| - `ddp_backend`: None |
| - `tpu_num_cores`: None |
| - `tpu_metrics_debug`: False |
| - `debug`: [] |
| - `dataloader_drop_last`: False |
| - `dataloader_num_workers`: 0 |
| - `dataloader_prefetch_factor`: None |
| - `past_index`: -1 |
| - `disable_tqdm`: False |
| - `remove_unused_columns`: True |
| - `label_names`: None |
| - `load_best_model_at_end`: True |
| - `ignore_data_skip`: False |
| - `fsdp`: [] |
| - `fsdp_min_num_params`: 0 |
| - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
| - `fsdp_transformer_layer_cls_to_wrap`: None |
| - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
| - `deepspeed`: None |
| - `label_smoothing_factor`: 0.0 |
| - `optim`: adamw_torch_fused |
| - `optim_args`: None |
| - `adafactor`: False |
| - `group_by_length`: False |
| - `length_column_name`: length |
| - `ddp_find_unused_parameters`: None |
| - `ddp_bucket_cap_mb`: None |
| - `ddp_broadcast_buffers`: False |
| - `dataloader_pin_memory`: True |
| - `dataloader_persistent_workers`: False |
| - `skip_memory_metrics`: True |
| - `use_legacy_prediction_loop`: False |
| - `push_to_hub`: True |
| - `resume_from_checkpoint`: None |
| - `hub_model_id`: None |
| - `hub_strategy`: every_save |
| - `hub_private_repo`: None |
| - `hub_always_push`: False |
| - `hub_revision`: None |
| - `gradient_checkpointing`: False |
| - `gradient_checkpointing_kwargs`: None |
| - `include_inputs_for_metrics`: False |
| - `include_for_metrics`: [] |
| - `eval_do_concat_batches`: True |
| - `fp16_backend`: auto |
| - `push_to_hub_model_id`: None |
| - `push_to_hub_organization`: None |
| - `mp_parameters`: |
| - `auto_find_batch_size`: False |
| - `full_determinism`: False |
| - `torchdynamo`: None |
| - `ray_scope`: last |
| - `ddp_timeout`: 1800 |
| - `torch_compile`: False |
| - `torch_compile_backend`: None |
| - `torch_compile_mode`: None |
| - `include_tokens_per_second`: False |
| - `include_num_input_tokens_seen`: False |
| - `neftune_noise_alpha`: None |
| - `optim_target_modules`: None |
| - `batch_eval_metrics`: False |
| - `eval_on_start`: False |
| - `use_liger_kernel`: False |
| - `liger_kernel_config`: None |
| - `eval_use_gather_object`: False |
| - `average_tokens_across_devices`: False |
| - `prompts`: None |
| - `batch_sampler`: no_duplicates |
| - `multi_dataset_batch_sampler`: proportional |
| - `router_mapping`: {} |
| - `learning_rate_mapping`: {} |
| |
| </details> |
| |
| ### Training Logs |
| | Epoch | Step | Training Loss | dot_ndcg@10 | |
| |:-------:|:---------:|:-------------:|:-----------:| |
| | 1.0 | 12500 | 11.5771 | 0.6587 | |
| | 2.0 | 25000 | 0.7888 | 0.6810 | |
| | 3.0 | 37500 | 0.7271 | 0.6884 | |
| | 4.0 | 50000 | 0.6774 | 0.6920 | |
| | 5.0 | 62500 | 0.6436 | 0.6912 | |
| | **6.0** | **75000** | **0.6274** | **0.6932** | |
|
|
| * The bold row denotes the saved checkpoint. |
|
|
| ### Framework Versions |
| - Python: 3.11.11 |
| - Sentence Transformers: 5.0.0 |
| - Transformers: 4.53.1 |
| - PyTorch: 2.6.0+cu124 |
| - Accelerate: 1.5.2 |
| - Datasets: 3.6.0 |
| - Tokenizers: 0.21.1 |
|
|
| ## Citation |
|
|
| ### BibTeX |
|
|
| #### Sentence Transformers |
| ```bibtex |
| @inproceedings{reimers-2019-sentence-bert, |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
| author = "Reimers, Nils and Gurevych, Iryna", |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
| month = "11", |
| year = "2019", |
| publisher = "Association for Computational Linguistics", |
| url = "https://arxiv.org/abs/1908.10084", |
| } |
| ``` |
|
|
| #### SpladeLoss |
| ```bibtex |
| @misc{formal2022distillationhardnegativesampling, |
| title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective}, |
| author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant}, |
| year={2022}, |
| eprint={2205.04733}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.IR}, |
| url={https://arxiv.org/abs/2205.04733}, |
| } |
| ``` |
|
|
| #### SparseMultipleNegativesRankingLoss |
| ```bibtex |
| @misc{henderson2017efficient, |
| title={Efficient Natural Language Response Suggestion for Smart Reply}, |
| author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, |
| year={2017}, |
| eprint={1705.00652}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL} |
| } |
| ``` |
|
|
| #### FlopsLoss |
| ```bibtex |
| @article{paria2020minimizing, |
| title={Minimizing flops to learn efficient sparse representations}, |
| author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s}, |
| journal={arXiv preprint arXiv:2004.05665}, |
| year={2020} |
| } |
| ``` |
|
|
| <!-- |
| ## Glossary |
|
|
| *Clearly define terms in order to be accessible across audiences.* |
| --> |
|
|
| <!-- |
| ## Model Card Authors |
|
|
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| --> |
|
|
| <!-- |
| ## Model Card Contact |
|
|
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| --> |
