---
license: apache-2.0
language:
- en
tags:
- mamba
- mamba2
- biomedical
- pubmed
- pubmedqa
- bioasq
- medqa
- sft
library_name: mamba-ssm
base_model: zmzfpc/biomamba-1.3b
datasets:
- qiaojin/PubMedQA
---

# BioMamba-BioMedQA-SFT-1.3b

BioMamba-1.3b supervised-finetuned for biomedical Yes/No/Maybe question
answering. The base model is the CPT checkpoint
[`zmzfpc/biomamba-1.3b`](https://huggingface.co/zmzfpc/biomamba-1.3b).

## Training data (SFT)

- **PubMedQA** (`qiaojin/PubMedQA`, `pqa_labeled` split) — MIT
- **BioASQ** (Yes/No subset, BioASQ 7b + 13b combined, locally mixed) — BioASQ DUA, non-commercial research use only

Instruction format: biomedical research question → answer **yes / no / maybe**.

## Evaluation

We evaluate on **PubMedQA**, **BioASQ (Yes/No)**, and **MedQA (USMLE 4-option)**.
**MedQA is not included in the SFT data** — the model is run on MedQA in a
transfer (non-in-distribution) setting, which is why this family is named
`biomedqa` rather than `medqa`.

## Loading

The checkpoint is saved in **mamba-ssm native format**, load with `mamba-ssm`:

```python
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
from transformers import AutoTokenizer

model = MambaLMHeadModel.from_pretrained("zmzfpc/biomamba-biomedqa-sft-1.3b", device="cuda", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("zmzfpc/biomamba-biomedqa-sft-1.3b")
```

`AutoModelForCausalLM.from_pretrained` will **not** work on this config.

## Intended use

Biomedical Yes/No/Maybe QA and transfer to MedQA-style multiple-choice.
Not intended for clinical decision-making and not validated on any
patient-facing task.

## Citation

Paper: **BioMamba: Domain-Adaptive Biomedical Language Models** ([arXiv:2408.02600](https://arxiv.org/abs/2408.02600)).
Code: https://github.com/LeoYML/BioMamba

```bibtex
@article{yue2024biomamba,
  title   = {{BioMamba}: Domain-Adaptive Biomedical Language Models},
  author  = {Yue, Ling and Zhu, Mingzhi and Xing, Sixue and Pan, Shaowu and
             Chenthamarakshan, Vijil and Wang, Yanbo and Cao, Yunning and
             Das, Payel and Fu, Tianfan},
  journal = {arXiv preprint arXiv:2408.02600},
  year    = {2024}
}
```