simcse-bert-base-uncased-snli-unsup

A sentence embedding model (768-d, [CLS] pooling) trained from bert-base-uncased by unsupervised SimCSE: the same sentence is encoded twice with different dropout masks (p=0.1); the two views are positives, other sentences in the batch are negatives. Course assignment U2T02 (Data Engineering, UPY), replicating SimCSE (Gao, Yao & Chen, EMNLP 2021). Author: Joel Perez.

Training data

The 165,529 unique sentences (premises and hypotheses) of a 100k-record subset of SNLI (records with annotator consensus only). This is a much smaller and narrower dataset than the one used in the paper (1M Wikipedia sentences for unsupervised; SNLI+MNLI for supervised), so the numbers are lower than the paper's.

Recipe

setting	value
base model	bert-base-uncased
pooling at inference	CLS (no MLP; a Linear+Tanh MLP was used on [CLS] during training only)
learning rate / batch size / epochs	3e-05 / 64 / 1
temperature τ	0.05
dropout	0.1
max sequence length (train / eval)	32 / 128
optimizer	AdamW, linear decay, no warm-up, grad-clip 1.0, mixed precision
seed (best of 3 by STS-B dev)	44 (checkpoint at step 500)
hardware	Tesla T4

Results (STS-B, Spearman ×100, cosine similarity of normalised embeddings, no regressor)

model	dev	test
this model	77.34	69.23
bert-base-uncased (mean)	59.31	47.29
SBERT-2019	80.77	76.98
SimCSE-unsup (paper)	82.50	76.85

The checkpoint was selected on the dev set; the test set was evaluated once per finished run.

Usage

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Joel119/simcse-bert-base-uncased-snli-unsup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."])
print(model.similarity(emb[0], emb[1]))

Limitations

• Trained on a small SNLI subset (image-caption domain, short sentences): it generalises less well to other domains/long texts than models trained on larger corpora.
• The model is English-only and was only evaluated on STS-B; other STS/SICK/transfer tasks were not tested.
• Sensitive to negation, antonyms and numbers (failure cases are discussed in the project report).
• It inherits the biases of BERT and of SNLI (crowd-sourced captions).
• Not a replacement for the original SimCSE checkpoints trained on the paper's data.