ModernBERT-base-mask-finetuned-shakespeare

This model is a fine-tuned version of answerdotai/ModernBERT-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.2340

How to use

You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:

import torch
from transformers import pipeline
from pprint import pprint

pipe = pipeline(
    "fill-mask",
    model="2nji/ModernBERT-base-mask-finetuned-shakespeare",
    torch_dtype=torch.bfloat16,
)

input_text = "Thou [MASK] on [MASK]."
results = pipe(input_text)
pprint(results)


<!-- [[{'score': 0.71875,
   'sequence': '[CLS]Thou art on[MASK].[SEP]',
   'token': 1445,
   'token_str': ' art'},
  {'score': 0.1416015625,
   'sequence': '[CLS]Thou hast on[MASK].[SEP]',
   'token': 16579,
   'token_str': ' hast'},
  {'score': 0.014892578125,
   'sequence': '[CLS]Thou be on[MASK].[SEP]',
   'token': 320,
   'token_str': ' be'},
  {'score': 0.00701904296875,
   'sequence': '[CLS]Thou Art on[MASK].[SEP]',
   'token': 3975,
   'token_str': ' Art'},
  {'score': 0.0042724609375,
   'sequence': '[CLS]Thou call on[MASK].[SEP]',
   'token': 1067,
   'token_str': ' call'}],
 [{'score': 0.1767578125,
   'sequence': "[CLS]Thou[MASK] on't.[SEP]",
   'token': 626,
   'token_str': "'t"},
  {'score': 0.146484375,
   'sequence': '[CLS]Thou[MASK] on me.[SEP]',
   'token': 479,
   'token_str': ' me'},
  {'score': 0.0419921875,
   'sequence': '[CLS]Thou[MASK] on it.[SEP]',
   'token': 352,
   'token_str': ' it'},
  {'score': 0.0419921875,
   'sequence': '[CLS]Thou[MASK] on earth.[SEP]',
   'token': 6149,
   'token_str': ' earth'},
  {'score': 0.03955078125,
   'sequence': '[CLS]Thou[MASK] on him.[SEP]',
   'token': 779,
   'token_str': ' him'}]] -->

Training and evaluation data

This model was finetuned using the the Shakespare_corpus Dataset

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	197	2.3128
No log	2.0	394	2.2150
2.3002	3.0	591	2.2395

Framework versions

• Transformers 4.48.3
• Pytorch 2.5.1+cu124
• Datasets 3.3.2
• Tokenizers 0.21.0