Instructions to use GiacomoSignorile/PatentBert-FineTuned with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use GiacomoSignorile/PatentBert-FineTuned with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("GiacomoSignorile/PatentBert-FineTuned")

sentences = [
    "COMPOSITIONS COMPRISING OR CONSISTING OF POLYDATIN FOR USE IN TREATMENT OF BONE DISEAS. Title: COMPOSITIONS COMPRISING OR CONSISTING OF POLYDATIN FOR USE IN TREATMENT OF BONE DISEAS\n\nAbstract: \n\nBusiness Description: The patent concerns the possible use of polydatin for the treatment of pathologies characterized by reduced bone mass. Polydatin is an oligostilbene that can be extracted in abundant quantities from the roots of the Polygonum Cuspidatum plant and, according to our studies about osteoblastic proliferation and differentiation, is able to increase osteogenic activity.\n\nTech Features: Skeletal pathologies are widespread in the world population, more than 200 million people are osteoporosis patients and have a high probability of fracture. Polydatin is a substance of natural origin, free of the side effects related to the use of synthetic drugs, currently administered for the treatment of bone loss diseases . Our experiments carried out in two-dimensional cultures (2D) and in cells grown on Scaffold (3D), have shown that polydatin is able to increase the activity of osteoblasts (ALP) and the deposition of Bone Matrix. The compositions comprising or consisting of polydatin may be administered orally (for example capsules or tablets, solutions, emulsions) or topically (oils, creams, ointments), or even through an appropriate enrichment of the active ingredient in functional foods\n\nApplications: Patients with traumatic-degenerative pathologies of the skeleton and oral cavity; Osteoporosis patients;\n\nAdvantages: Molecule of natural origin; Antioxidant activity on the whole organism; Possibility of topical use;",
    "Inhibitory compounds neurodegeneration and tumors. Title: Inhibitory compounds neurodegeneration and tumors\n\nAbstract: \n\nBusiness Description: The inhibition of the monoacylglycerol lipase enzyme (MAGL), naturally present in many brain cells and involved in physio-pathological processes, has a high therapeutic potential: neurodegenerative inflammation pathologies and tumors could be treated with new reversible inhibitory compounds, which would reduce the side effects of the irreversible inhibitors tested so far.\n\nTech Features: Monoacylglycerol lipase (MAGL) is a human enzyme of the endocannabinoid system involved in numerous physio-pathological processes (regulation of inflammation, anxiety, immune modulation, motor coordination ...), yet its overexpression/upregulation can cause neuroinflammatory diseases and tumors. The inhibition of MAGL for therapeutic purposes has been studied so far with irreversible inhibitors, which however nullify the enzyme activity, leading to a progressive loss of the therapeutic effect and to addiction phenomena. On the contrary, the new-patented compounds based on a strong non-covalent reversible mechanism of action avoid the side effects mentioned. Effective in laboratory on various tumor cell lines (e.g. colorectal, breast and ovarian cancer), they could also treat other MAGL-mediated pathologies (neuroinflammation/degeneration, pain, amyotrophic multiple/lateral sclerosis, Alzheimer's disease, Parkinson's disease). Link to scientific publication and information on Ca' Foscari website\n\nApplications: Innovative and less harmful pharmaceutical compositions for the treatment of serious neurodegenerative pathological conditions; Innovative and less harmful pharmaceutical compositions for cancer treatment.\n\nAdvantages: Temporary nature and reversibility; Drastic reduction of side effects; High efficacy tested on tumor cell lines; Exploitable to treat numerous neurodegenerative diseases; One of the few non-covalent reversible MAGL inhibitors with high efficacy.",
    "New computational method for the prognosis of amyotrophic lateral sclerosis. Title: New computational method for the prognosis of amyotrophic lateral sclerosis\n\nAbstract: A method is described for determining a disease progression and survival prognosis, at a succession of prediction times, for patients suffering from amyotrophic lateral sclerosis (ALS). The method comprises a step of defining a set of variables associated with the onset and progression of amyotrophic lateral sclerosis, comprising a first group of variables associated with the onset of amyotrophic lateral sclerosis (comprising at least the variables “patient sex”, “disease onset age”, “disease onset site”), a second group of dynamic time variables (comprising at least the variable “time elapsed since disease onset”), a third group of dynamic functional variables (comprising at least one of the variables breathing, swallowing, communicating, walking/self-care or at least one variable of a functional progression and/or severity scale of amyotrophic lateral sclerosis), and further at least one variable associated with survival. The method further provides for encoding by means of a Dynamic Bayesian Network, using at least one trained algorithm, a plurality of probabilistic conditional dependence relationships, in which each relationship is a probabilistic conditional dependence relationship between two of the aforesaid variables. The aforesaid prediction times are defined so that each prediction time belongs to a respective time interval in which the conditional dependence relationships between the variables are stationary. The method further involves describing the Dynamic Bayesian Network, using at least one trained algorithm, by means of a corresponding graph, comprising said variables as nodes and comprising topological connections oriented between nodes corresponding to variables among which a probabilistic conditional dependence is identified. In the graph, given a node, the connections entering it show a conditional probability of the value assumed by the variable associated with such node, in a given prediction time, depending on the values assumed, in a prior prediction time, from the variables associated with the nodes from which such connections originate. The method further comprises the steps of entering, for each of the defined variables, data acquired at a given acquisition time relating to the situation of a specific patient; and calculating, by electronic processing and/or calculating means, on the basis of the Dynamic Bayesian Network and the graph, and starting from the aforesaid acquired data, the values of each of the defined variables, at one or more prediction times following the acquisition time. Finally, the method involves obtaining, in a given prediction time, disease progression prognosis results on the basis of the values of one or more of the variables of the third group calculated in such prediction time; and the survival prognosis results on the basis of the value of at least one variable associated with survival, calculated at such prediction time.\n\nBusiness Description: The invention relates to a new probabilistic model based on dynamic Bayesian networks: by introducing specific clinical data of patients suffering from amyotrophic lateral sclerosis (ALS), the model object of the invention allows to predict the course of the disease, the loss of autonomy in specific functional domains and the survival, and to identify risk factors of the disease.\n\nTech Features: Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disease, characterized by progressive muscle paralysis caused by motor neuron degeneration. The onset and progression of ALS are very heterogeneous, making the development of personalised approaches extremely difficult. The invention consists of a probabilistic predictor of ALS progression, based on a dynamic Bayesian network built using a database of over 4000 patients treated at specialised centres in Italy and Israel. By intoducing the clinical information of a new ALS subject, the invention allows to simulate the clinical course and to probabilistically predict the risk of impairment of the functional domains typical of ALS and the survival, representing an important tool to support clinical decision-making and the drafting of the treatment plan.\n\nApplications: Online platform (SaaS)/software for predicting ALS progression; Support to clinical decision and drafting of the treatment plan; Institutional stakeholder decision support; In silico generation of patient populations with specific characteristics.\n\nAdvantages: Prediction of ALS progression; Personalized medicine approach; Clinical decision support; Identification of new prognostic markers; Possible alternative to placebo cohorts in clinical trials.",
    "COMPOSITIONS COMPRISING OR CONSISTING OF POLYDATIN FOR USE IN TREATMENT OF BONE DISEAS. Title: COMPOSITIONS COMPRISING OR CONSISTING OF POLYDATIN FOR USE IN TREATMENT OF BONE DISEAS\n\nAbstract: \n\nBusiness Description: The patent concerns the possible use of polydatin for the treatment of pathologies characterized by reduced bone mass. Polydatin is an oligostilbene that can be extracted in abundant quantities from the roots of the Polygonum Cuspidatum plant and, according to our studies about osteoblastic proliferation and differentiation, is able to increase osteogenic activity.\n\nTech Features: Skeletal pathologies are widespread in the world population, more than 200 million people are osteoporosis patients and have a high probability of fracture. Polydatin is a substance of natural origin, free of the side effects related to the use of synthetic drugs, currently administered for the treatment of bone loss diseases . Our experiments carried out in two-dimensional cultures (2D) and in cells grown on Scaffold (3D), have shown that polydatin is able to increase the activity of osteoblasts (ALP) and the deposition of Bone Matrix. The compositions comprising or consisting of polydatin may be administered orally (for example capsules or tablets, solutions, emulsions) or topically (oils, creams, ointments), or even through an appropriate enrichment of the active ingredient in functional foods\n\nApplications: Patients with traumatic-degenerative pathologies of the skeleton and oral cavity; Osteoporosis patients;\n\nAdvantages: Molecule of natural origin; Antioxidant activity on the whole organism; Possibility of topical use;"
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]

Notebooks
Google Colab
Kaggle

PatentBert-FineTuned / README.md

GiacomoSignorile

Initial upload of fine-tuned PatentSBERTa Specialist (eHealth)

8e441ee verified 7 months ago

preview code

Raw History Blame

11.9 kB

metadata

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:2633
  - loss:MultipleNegativesRankingLoss
  - dataset_size:7861
base_model: AI-Growth-Lab/PatentSBERTa
widget:
  - source_sentence: CONSORTIUM OF PROBIOTICS.
    sentences:
      - NANOSTRUCTURED FILTERING MEMBRANE FOR ANALYTICAL APPLICATIONS.
      - CONSORTIUM OF PROBIOTICS.
      - Real-time communications over Bluetooth Low Energy.
  - source_sentence: Device for the collection and analysis of a biological fluid.
    sentences:
      - Device for the collection and analysis of a biological fluid.
      - STED MICROSCOPY BASED ON SYNCHRONOUS DETECTION OF FLUORESCENCE EMISSION.
      - Robotic apparatus for minimally invasive surgery.
  - source_sentence: Method for obtaining a porous semiconductor.
    sentences:
      - 'Racetrack Logic: smart in-memory computing.'
      - Method for obtaining a porous semiconductor.
      - Salts of benzimidazole compounds, their use and synthetic preparation.
  - source_sentence: Enthalpy exchangers with polymeric membrane.
    sentences:
      - Closure system for engine connecting rods.
      - Enthalpy exchangers with polymeric membrane.
      - Soluble protein with high angiogenic activity.
  - source_sentence: FLEXIBLE CAPACITIVE KEYBOARD FOR TEXTILE USE.
    sentences:
      - 'GreenValve III: System for spherical segment valves.'
      - FLEXIBLE CAPACITIVE KEYBOARD FOR TEXTILE USE.
      - Mandibular advancement device.
pipeline_tag: sentence-similarity
library_name: sentence-transformers

SentenceTransformer based on AI-Growth-Lab/PatentSBERTa

This is a sentence-transformers model finetuned from AI-Growth-Lab/PatentSBERTa. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: AI-Growth-Lab/PatentSBERTa
Maximum Sequence Length: 384 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False, 'architecture': 'MPNetModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'FLEXIBLE CAPACITIVE KEYBOARD FOR TEXTILE USE.',
    'FLEXIBLE CAPACITIVE KEYBOARD FOR TEXTILE USE.',
    'GreenValve III: System for spherical segment valves.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.3686],
#         [1.0000, 1.0000, 0.3686],
#         [0.3686, 0.3686, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 7,861 training samples
Columns: sentence_0 and sentence_1
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1
type string string
details
min: 4 tokens
mean: 12.32 tokens
max: 32 tokens

min: 8 tokens
mean: 12.01 tokens
max: 36 tokens

	sentence_0	sentence_1
type	string	string
details	min: 4 tokens mean: 12.32 tokens max: 32 tokens	min: 8 tokens mean: 12.01 tokens max: 36 tokens

Samples:

sentence_0	sentence_1
`Genetic diagnostic technique for SCA1-3,6,7`	`Applicant/Organization: UNIV DEGLI STUDI DI TORINO`
`Method for detecting Macrophomina phaseolina`	`Technical Classification: C12Q`
`System for controlled administration of a substance from a human -body-implanted infusion device`	`Applicant/Organization: Stefanini Cesare`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 12
per_device_eval_batch_size: 12
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

per_device_train_batch_size: 12
num_train_epochs: 3
max_steps: -1
learning_rate: 5e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1
label_smoothing_factor: 0.0
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: trackio
eval_strategy: no
per_device_eval_batch_size: 12
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: False
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_backend: None
ddp_timeout: 1800
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
1.5152	500	0.0002
3.0303	1000	0.0000
4.5455	1500	0.0000
0.7622	500	2.3552
1.5244	1000	1.9437
2.2866	1500	1.8059

Framework Versions

Python: 3.12.7
Sentence Transformers: 5.2.3
Transformers: 5.3.0
PyTorch: 2.10.0
Accelerate: 1.13.0
Datasets: 4.2.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}