File size: 3,007 Bytes
4443fda | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
language: en
license: apache-2.0
---
# BiomedBERT Hash Nano
This is a `970K` parameter [BERT](https://arxiv.org/abs/1810.04805) encoder-only model trained on data from [PubMed](https://pubmed.ncbi.nlm.nih.gov/). The raw data was transformed using [PaperETL](https://github.com/neuml/paperetl) with the results stored as a local dataset via the [Hugging Face Datasets library](https://huggingface.co/docs/datasets/en/index).
`biomedbert-hash-nano` is built with the BERT Hash architecture as described in the following links.
- [BERT Hash Nano](https://huggingface.co/NeuML/bert-hash-nano)
- [Training Tiny Language Models with Token Hashing](https://medium.com/neuml/training-tiny-language-models-with-token-hashing-b744aa7eb931)
## Usage
`biomedbert-hash-nano` can be loaded using [Hugging Face Transformers](https://huggingface.co/docs/transformers/en/inde) as follows. Note that given that this is a custom architecture, `trust_remote_code` needs to be set.
```python
from transformers import AutoModel
model = AutoModel.from_pretrained("neuml/biomedbert-hash-nano", trust_remote_code=True)
```
The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.
## Evaluation Results
This [Medical Abstracts Text Classification Dataset](https://huggingface.co/datasets/TimSchopf/medical_abstracts) was used to evaluate the model's performance. A handful of biomedical models and general models were selected for comparison.
Metrics were generated using Hugging Face's standard [run_glue script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py) as shown below.
```bash
python run_glue.py --model_name_or_path neuml/biomedbert-hash-nano --dataset-name medclassify --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 1e-4 --num_train_epochs 4 --output_dir outputs --trust-remote-code True
```
_Note: The original dataset was saved locally as `medclassify` the the `condition_label` column renamed to `label` to work more easily with the glue script_
| Model | Parameters | Accuracy | Loss |
| ----- | ---------- | --------------- | ---------------- |
| [**biomedbert-hash-nano**](https://hf.co/neuml/biomedbert-hash-nano) | **0.969M** | **0.6195** | **0.9464** |
| [bert-hash-nano](https://hf.co/neuml/bert-hash-nano) | 0.969M | 0.5045 | 1.2192 |
| [bert-base-uncased](https://hf.co/google-bert/bert-base-uncased) | 110M | 0.6118 | 0.9712 |
| [biomedbert-base](https://hf.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) | 110M | 0.6195 | 0.9037 |
| [ModernBERT-base](https://hf.co/answerdotai/ModernBERT-base) | 149M | 0.5672 | 1.1079 |
| [BioClinical-ModernBERT-base](https://hf.co/neuml/thomas-sounack/BioClinical-ModernBERT-base) | 149M | 0.5679 | 1.0915 |
As we can see, this model performs very well against models much larger in size. This dataset is a challenging one!
|