BERT-NLI-vaccine-v2

A DeBERTaV3-large model fine-tuned on the Recognizing Textual Entailment (RTE) task to automatically classify vaccine-related news headlines into 10 thematic categories.

This model was developed as part of the study:

Neves, L. F. F., Camargo, C. Q., & Massarani, L. (2025). Boosting Transformers: Recognizing Textual Entailment for Classification of Vaccine News Coverage. Computational Communication Research, 7(1), 1–32. https://doi.org/10.5117/CCR2025.1.1.NEVE

Code and training data are available at: https://github.com/lffernandes08/ccr_codes

Model description

The model follows the entailment-as-few-shot-learner approach (Wang et al., 2021; Laurer et al., 2023): each news headline (the text, T) is paired with a natural-language description of a topic class (the hypothesis, H), and the model predicts whether the headline entails, contradicts, or is neutral with respect to that description. Classification is performed by testing the headline against all 10 class descriptions and selecting the one with the highest entailment score.

The model is a fine-tuned version of MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-lingwanli, further trained on 800 manually annotated vaccine-related headlines.

Intended uses

This model is intended for researchers in science communication, computational social science, and related fields who need to classify vaccine-related news content into the 10 categories below — either by reusing these exact categories or by adapting the approach to new, custom classes expressed as natural-language descriptions.

It is not a general-purpose vaccine safety, fact-checking, or sentiment classifier: it does not judge whether a claim about vaccines is true, false, or dangerous. It classifies the topic/theme a news headline is about.

The 10 classes

Class	Description
Global Access to Vaccine	Initiatives for equal access to COVID-19 vaccines (e.g., COVAX), and/or the need to combat inequity, ensure global distribution, and contain the disease, especially in low- and middle-income countries
Science and Technology	The science behind vaccine development, including research, studies, safety and efficacy trials, emergency approval, and side effects
Public Health Policies	Vaccine mandates, vaccine passports, and/or other public health policies such as social distancing, quarantine, lockdowns, and mask mandates
Vaccination Rollout/Campaign	Vaccination rollout, campaigns, priority groups, vaccine deals, and/or the monitoring of vaccination rates
Vaccine Hesitancy and Mis-/Disinformation	The circulation of disinformation and misinformation regarding vaccines, which contribute to vaccine hesitancy and reluctance
Public Endorsement to Vaccine	Public endorsement and incentive for vaccines, such as celebrities and artists getting vaccinated
Institutional Affairs	Institutional and governmental issues, political disputes, conflicts, political authorities, and export bans
Problems in Vaccination	Problems in vaccination, such as delays, fraud, disparities, vaccine shortages
Public Perception of Vaccine	Opinion polls and surveys on the public perception of vaccines and/or willingness to get vaccinated
Economic Consequences	The impacts and benefits of vaccination on the economy

How to use

from transformers import pipeline

pipe = pipeline("text-classification", model="LuizNeves/BERT-NLI-vaccine-v2")

classes = {
    "Global Access to Vaccine": "The headline addresses initiatives for equal access to covid-19 vaccines, such as COVAX, and/or the need to combat inequity, ensure global distribution, and contain the disease, especially to low and middle-income countries",
    "Science and Technology": "The headline addresses the science behind vaccine development, including research, studies, safety and efficacy trials, emergency approval, and side effects",
    "Public Health Policies": "The headline addresses vaccine mandates, vaccine passports, and/or other public health policies such as social distancing, quarantine, lockdowns, and mask mandates",
    "Vaccination Rollout/Campaign": "The headline addresses vaccination rollout, campaigns, priority groups, vaccine deals, and/or the monitoring of vaccination rates",
    "Vaccine Hesitancy and Mis-/Disinformation": "The headline addresses the circulation of disinformation and misinformation regarding vaccines, which contribute to vaccine hesitancy and reluctance",
    "Public Endorsement to Vaccine": "The headline addresses public endorsement and incentive for vaccines, such as celebrities and artists getting vaccinated",
    "Institutional Affairs": "The headline addresses institutional and governmental issues, political disputes, conflicts, political authorities, and export bans",
    "Problems in Vaccination": "The headline addresses problems in vaccination, such as delays, fraud, disparities, vaccine shortages",
    "Public Perception of Vaccine": "The headline addresses opinion polls and surveys on the public perception of vaccines and/or willingness to get vaccinated",
    "Economic Consequences": "The headline addresses the impacts and benefits of vaccination on the economy",
}

headline = "Twitter will begin removing vaccine misinformation"

for class_name, description in classes.items():
    result = pipe({"text": headline, "text_pair": description})
    print(f"{class_name}: {result['label']} ({result['score']:.2%})")

# The class with the "entailment" label (highest score) is the predicted category.

Training data

The model was fine-tuned on a manually annotated corpus of 1,000 vaccine-related news headlines, published by four elite newspapers (The New York Times, The Guardian, China Daily, and Folha de S.Paulo) between 1 January 2020 and 31 December 2021. Headlines were annotated by a science communication researcher and validated by a second coder (intercoder reliability: Cohen's Kappa / Krippendorff's Alpha = 0.825). Classes are unevenly distributed (from 51 to 143 headlines per class).

Following the protocol of Laurer et al. (2023), each true headline-description pair used for training was paired with a randomly sampled false pair, doubling the effective training set and teaching the model to discriminate between true and false entailment.

This model version was fine-tuned on 800 of the 1,000 annotated headlines (the remaining 200 were held out as the test set).

Evaluation results

Evaluated on a held-out test set of 200 headlines:

Metric	Score
Accuracy / F1 Micro	0.930
Balanced Accuracy / Recall Macro	0.924
F1 Macro	0.927

This was the best-performing configuration among eight models and procedures compared in the original study (fine-tuned Transformers, prompt-based GPT models, and off-the-shelf RTE models), outperforming both closed-source GPT-based zero/few-shot classification and non-fine-tuned open-source RTE models (see Table 5 in the paper).

Limitations

• The model was trained exclusively on English-language headlines (Portuguese-language headlines in the original corpus were translated into English prior to training). Performance on non-English text has not been evaluated.
• The training and test sets are relatively small (1,000 and 200 headlines, respectively), which limits the precision of the reported evaluation metrics, particularly for minority classes.
• The 10 classes were defined for a specific research context (COVID-19 vaccine coverage, 2020–2021) and may not capture topics relevant to other periods or vaccine-related contexts without adaptation.
• As with any model built on pre-trained language representations, it may carry biases present in its training data.

Citation

If you use this model, please cite the original paper:

@article{neves2025boosting,
  title={Boosting Transformers: Recognizing Textual Entailment for Classification of Vaccine News Coverage},
  author={Neves, Luiz F. F. and Camargo, Chico Q. and Massarani, Luisa},
  journal={Computational Communication Research},
  volume={7},
  number={1},
  pages={1--32},
  year={2025},
  doi={10.5117/CCR2025.1.1.NEVE}
}