gianvr/sbert-tunned-covid

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Finetuning

The model was finetuned on Coronavirus tweets NLP - Text Classification. The objective was agrupate tweets by sentiment, for this, the dataset was divided into pairs of tweets and the similarity between them defined by the sentiment label. The following labels were used:

• Extremely Positive and Extremely Negative: 0.0
• Extremely Positive and Neutral: 0.2
• Extremely Positive and Positive: 0.7
• Extremely Positive and Negative: 0.2
• Extremely Negative and Neutral: 0.2
• Extremely Negative and Positive: 0.2
• Extremely Negative and Negative: 0.7
• Neutral and Positive: 0.5
• Neutral and Negative: 0.5

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("gianvr/sbert-tunned-covid")
# Run inference
sentences = [
    'Governor Cuomo Please order retail businesses to close My husband has to go to work and expose himself to the virus Store owners will not close unless you tell them they have to I have two friends fighting for their lives PLEASE CLOSE STORES NOW',
    'Clothier @LandsEnd has closed all of its retail stores through March 29th due to coronavirus pandemic.   They will pay all of their employees during the time period.\r\r\n\r\r\nThis is their store at the Hunt Valley Towne Center.  #HuntValley #coronavirus https://t.co/DIwNzAOLeb',
    'Went to the store today. Most canned food is gone. We needed toilet paper and actually got a pack of 12 rolls. We got milk and cat treats as well. Thank god for Save on Foods for still having some stuff. \r\r\nOver 100 cases of Covid-19 in my province so panic buying is common.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Coronavirus tweets NLP - Text Classification

• Size: 59,616 training samples

• Columns: sentence1, sentence2, and label

• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	label
  type	string	string	float
  details	min: 11 tokens mean: 60.62 tokens max: 245 tokens	min: 6 tokens mean: 58.31 tokens max: 245 tokens	min: 0.0 mean: 0.36 max: 0.7

• Samples:

  sentence1	sentence2	label
  BBC Business correspondent just now: "We're going to see a sustained demand for food" Pretty sure that's not news?#CoronaCrisis #covid19uk	A warning from Nick Talley, University of Newcastle professor and neuro-gastroenterologist, for those who need to hear it: https://t.co/UzRRswUSTS	0.7
  The announcement(over the PA)at #Woolworths #Marrickville Metro supermarket telling shoppers to maintain #distancing is absurd. In a supermarket aisle there just isn't enough room! #Auspol #NSWpol #coronavirus #Covid_19australia #COVID19Aus	Due to the closure of indoor shopping malls to prevent the spread of our Brooklyn Atlantic Office will be closed until further notice We apologize for the inconvenience Please visit our website to find more than 60 online transactions	0.5
  Expert s top tips for supermarket shopping 1 Avoid cash 2 Use a key to type in credit card pin 3 Don t take a receipt 4 Use hand sanitizer once back at your car 5 Consider quarantining food Read more	Covid-19 is likely to disrupt these data in the coming months. So far, there is only limited evidence of surging demand in supermarkets stoking prices (mostly non-food and bread). Meanwhile, fruit, veg, and sugary item prices have all crashed. 2/3	0.2

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Training Hyperparameters

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3.0
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss
0.0671	500	0.1053
0.1342	1000	0.0802
0.2013	1500	0.0741
0.2684	2000	0.0701
0.3355	2500	0.0682
0.4026	3000	0.0665
0.4697	3500	0.0674
0.5368	4000	0.0667
0.6039	4500	0.0624
0.6710	5000	0.0613
0.7381	5500	0.0589
0.8052	6000	0.0647
0.8722	6500	0.0645
0.9393	7000	0.0626
1.0064	7500	0.0616
1.0735	8000	0.0591
1.1406	8500	0.0601
1.2077	9000	0.0592
1.2748	9500	0.0562
1.3419	10000	0.0586
1.4090	10500	0.0593
1.4761	11000	0.0578
1.5432	11500	0.0574
1.6103	12000	0.0561
1.6774	12500	0.0579
1.7445	13000	0.0556
1.8116	13500	0.0551
1.8787	14000	0.0554
1.9458	14500	0.0537
2.0129	15000	0.0548
2.0800	15500	0.0521
2.1471	16000	0.0534
2.2142	16500	0.0529
2.2813	17000	0.0529
2.3484	17500	0.052
2.4155	18000	0.0527
2.4826	18500	0.0517
2.5497	19000	0.0517
2.6167	19500	0.0497
2.6838	20000	0.0501
2.7509	20500	0.0516
2.8180	21000	0.0515
2.8851	21500	0.0511
2.9522	22000	0.0489

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.2.1
• Transformers: 4.44.2
• PyTorch: 2.5.0+cu121
• Accelerate: 0.34.2
• Datasets: 3.1.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}