AmirMohseni/tfidf-logreg-v3-user-seed42

TF-IDF + logistic-regression cascade for legal-guidance conversations

This repository contains the frozen full-conversation lexical baseline for the Legal Guidance in the Wild study. It first predicts whether the user is seeking legal guidance and, for predicted-positive conversations, assigns one of 14 primary legal topics. The models receive chronological user messages only; assistant responses are excluded.

This is a research classifier, not a legal-advice system. Its output must not be treated as a determination of a person's rights, claims, or legal needs.

Artifacts

• guidance_tfidf_logreg.joblib: fitted binary guidance model, vectorizers, label order, and selected threshold
• topic_tfidf_logreg.joblib: fitted 14-way topic model, vectorizers, and topic names
• guidance_metrics.json, topic_metrics.json, and cascade_val_metrics.json: complete silver-validation metrics
• majority_metrics.json: majority-baseline results
• validation_predictions.csv: all 290 frozen validation predictions
• run_metadata.json: data fingerprints, software versions, and search space

Model design

Each stage concatenates word unigram/bigram TF-IDF features (maximum 150,000) with character-within-word 3--5-gram TF-IDF features (maximum 200,000), then fits class-balanced logistic regression. The regularization value was selected from C = {0.25, 1, 4}; both stages selected C = 1. The guidance threshold was selected on the silver validation split and is stored in the artifact (0.51 for this run).

Data and reproducibility

• Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
• Requested dataset revision: main (latest at run time)
• Train fingerprint: 6ba4c2696e889276
• Validation fingerprint: 403fe118d76c8360
• Train / validation conversations: 1,632 / 290
• Topic-stage train / validation rows: 750 / 134 guidance-positive conversations
• Input mode: chronological user messages only
• Seed: 42
• scikit-learn: 1.6.1

Dataset access follows the sharing settings of the linked dataset repository. The fingerprints above identify the exact loaded splits even if main later changes.

Silver-validation results

Stage	Metric	Value
Guidance (N=290)	Macro-F1	0.8653
Guidance (N=290)	Positive-class F1	0.8602
Guidance (N=290)	Accuracy	0.8655
Guidance (N=290)	Balanced accuracy	0.8676
Guidance (N=290)	AUPRC	0.9105
Topic, guidance-positive only (N=134)	Accuracy	0.6567
Topic, guidance-positive only (N=134)	Macro-F1	0.4927
End-to-end 15-way cascade (N=290)	Accuracy	0.7310
End-to-end 15-way cascade (N=290)	Macro-F1	0.4514

At threshold 0.51, the guidance confusion matrix is TN=131, FP=25, FN=14, TP=120. These are silver-validation results used for development and model selection; they are not final adjudicated-gold estimates.

Loading the cascade

Install compatible dependencies:

pip install "scikit-learn==1.6.1" joblib

The following shows the artifact structure and cascade logic. texts must use the same chronological, user-only serialization as training.

import joblib
from scipy.sparse import hstack

texts = ["My landlord is withholding my deposit. What can I do?"]

guidance = joblib.load("guidance_tfidf_logreg.joblib")
Xg = hstack([
    guidance["word_vectorizer"].transform(texts),
    guidance["char_vectorizer"].transform(texts),
], format="csr")
guidance_probability = guidance["model"].predict_proba(Xg)[:, 1]
seeks_guidance = guidance_probability >= guidance["threshold"]

topic = joblib.load("topic_tfidf_logreg.joblib")
Xt = hstack([
    topic["word_vectorizer"].transform(texts),
    topic["char_vectorizer"].transform(texts),
], format="csr")
topic_id = topic["model"].predict(Xt)
topic_name = [topic["topic_names"][int(i)] for i in topic_id]

cascade_prediction = [
    name if positive else "NO_GUIDANCE"
    for positive, name in zip(seeks_guidance, topic_name)
]

joblib uses pickle-based serialization. Load artifacts only from a repository and revision you trust.

Limitations

The models were trained on English-language public LLM interaction logs with silver labels, one data source, and one random seed. Rare topics have very small validation support, and three topics have zero validation F1 in this run. The dataset is jurisdiction-agnostic, may contain sensitive material, and is not representative of all people who seek legal help. Threshold calibration and performance may shift in other platforms, jurisdictions, time periods, or deployment populations. Human review is required for consequential use.

Related repositories

The study also releases full-conversation ModernBERT-base guidance and topic checkpoints, full-conversation ModernBERT-large guidance and topic checkpoints, and prefix-aware base, large, and TF-IDF models.

Citation

Please cite the accompanying Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations manuscript when it becomes available.