RadwaElashry2030/baseera-fake-review-ensemble

Baseera Fake-Review Ensemble

A fake/deceptive-review detector built for the Baseera Olist-marketplace review-intelligence project, after two prior checkpoints were found to be unreliable and rejected. This model is the result of a full investigation, not a first attempt — see Motivation & investigation below.

Two components, used together (mean of their probabilities) or independently:

Component	Size	Role
distilbert/	~257MB	DistilBERT, fine-tuned with a paraphrase-consistency loss
tfidf/	~350KB	TF-IDF (1–2 grams) + Logistic Regression

Trained on the Ott et al. Deceptive Opinion Spam Corpus (Cornell, ACL 2011 / NAACL 2013) — 1,596 hotel reviews, genuinely human-verified deceptive-vs-truthful labels (deceptive reviews were written by Mechanical Turk workers explicitly instructed to write a convincing fake review; truthful reviews are real, sourced reviews). This is a genuine deceptive-intent label, not a proxy for star rating or any other confound.

Validated results

Measured over all 320 held-out test reviews (not a handful of hand-picked examples), with Wilson 95% confidence intervals on the flip rate under WordNet paraphrasing:

	Test accuracy	Raw flip rate	Abstain rate*	Confident flip rate
DistilBERT alone	91.25%	1.9% (CI 0.9–4.0%)	4.4%	0.7% (CI 0.2–2.4%)
TF-IDF alone	91.25%	5.9% (CI 3.8–9.1%)	41.2%	0.0% (CI 0–2.0%)
Ensemble (mean)	91.6%	1.2% (CI 0.5–3.2%)	6.2%	0.0% (CI 0–1.3%)

* "Abstain rate" = predictions within 0.5±0.1 of the decision boundary, reported as UNCERTAIN rather than a forced confident call (see Usage).

"Confident flip rate" = out of predictions where the model was confident on both the original and a WordNet-paraphrased version of the same review, how often did the two confident verdicts disagree. This is the number that matters for "can I trust a single confident verdict" — and it's ~0% for the ensemble, with a tight confidence interval, not a lucky handful of examples.

Motivation & investigation

Two prior checkpoints were tried and rejected before this one:

1. An external pretrained model (jb10231/fake-review-detector) — its label semantics were never wired into its published config, and a pure meaning-preserving synonym substitution flipped one verdict from 99.9% to 0.1% confidence.
2. A first retrain on a different dataset (AI-generated-vs-human-written text, 97% held-out accuracy) — failed the exact same paraphrase-stability test the same way. A good test-set score does not prove robustness to rewording.

A candidate replacement dataset (a large Amazon "spam/non-spam" corpus) was also rejected before any training was attempted: direct inspection showed its label was a 1:1 proxy for star rating (100% of 4–5★ reviews labeled "spam", 100% of 1–3★ labeled "not spam", zero overlap) — not a genuine spam judgment at all.

This model closes the gap two ways:

• Paraphrase-consistency training: a symmetric KL-divergence loss between each training review's prediction and a WordNet-paraphrased + length-perturbed view of the same review, added alongside the normal classification loss.
• Ensembling with TF-IDF+LogReg: a bag-of-words linear model has no positional/attention mechanism for a transformer's length-sensitivity to act through, so it's inherently more length-robust, at the cost of abstaining more often. Averaging the two closes both failure modes better than either alone.

Full write-up, including the two rejected checkpoints and the rejected dataset: MODEL_COMPARISON_AUDIT.md §9 in the main project repo.

Honest, unresolved limitation

Trained on Chicago hotel reviews, applied in the Baseera app to Olist e-commerce reviews. This is a real domain shift and has not been separately measured on e-commerce data — no genuinely-labeled fake-review dataset exists for that domain (the same gap that ruled out training directly on it in the first place). Treat any single verdict as a screening signal, not a validated fraud finding.

Usage

import joblib
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from huggingface_hub import snapshot_download

repo = "RadwaElashry2030/baseera-fake-review-ensemble"
local = snapshot_download(repo)

tokenizer = AutoTokenizer.from_pretrained(f"{local}/distilbert")
bert_model = AutoModelForSequenceClassification.from_pretrained(f"{local}/distilbert").eval()
vectorizer = joblib.load(f"{local}/tfidf/vectorizer.pkl")
clf = joblib.load(f"{local}/tfidf/classifier.pkl")

def score(text: str) -> float:
    enc = tokenizer(text, truncation=True, max_length=256, return_tensors="pt")
    with torch.no_grad():
        bert_prob = torch.softmax(bert_model(**enc).logits, dim=1)[0, 1].item()
    tfidf_prob = clf.predict_proba(vectorizer.transform([text]))[0, 1]
    return (bert_prob + tfidf_prob) / 2.0

def verdict(p: float, margin: float = 0.1) -> str:
    if p >= 0.5 + margin: return "FAKE"
    if p <= 0.5 - margin: return "REAL"
    return "UNCERTAIN"   # honestly reports ambiguity instead of forcing a guess

p = score("This product exceeded all my expectations, best purchase ever!")
print(verdict(p), round(p, 3))

RAM-constrained deployments: the tfidf/ component alone (~350KB) can be used without loading DistilBERT at all — independently measured at 0/188 confident flips (95% CI upper bound 2.0%), at the cost of a higher abstain rate (41.2% vs. the full ensemble's 6.2%). This is what the live Baseera deployment actually runs on its memory-constrained free-tier host.

Training

See the training scripts in the main project repo: train_fake_review_detector_v2_consistency.py and train_fake_review_detector_tfidf.py.

Base checkpoint: distilbert-base-uncased. Seed 42 throughout. 70/10/20 split by (label, sentiment polarity), zero text overlap verified.

Citation

If you use the underlying dataset, please cite the original authors:

M. Ott, Y. Choi, C. Cardie, and J.T. Hancock. 2011. Finding Deceptive Opinion Spam by Any
Stretch of the Imagination. ACL-HLT 2011.

M. Ott, C. Cardie, and J.T. Hancock. 2013. Negative Deceptive Opinion Spam. NAACL-HLT 2013.