TF-IDF + logistic regression for prefix-aware legal-guidance detection
This lightweight baseline detects whether legal guidance is already present in a cumulative user-turn prefix. It is a research baseline, not a legal-advice system, and must not be used to decide whether a person has a valid claim.
Method
The model combines word (1,2)-gram and char_wb (3,5)-gram TF-IDF features
with logistic regression. Prefixes before first_guidance_user_turn_id are
negative; the onset prefix and all subsequent prefixes are positive. Each
prefix is weighted by the inverse of the conversation's number of user turns,
so every conversation has total training weight 1.0.
Both the regularization parameter and probability threshold were selected on the silver validation split by conversation-weighted prefix macro-F1, with weighted positive-class F1 as the tie-breaker.
Data and configuration
- Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
- Requested revision:
main(latest at execution) - Resolved revision:
489b87c18e571e8c7e9a3b1c2463b79e50cef120 - Training: 1,632 conversations / 4,865 prefixes
- Validation: 290 conversations / 912 prefixes
- Selected
C: 0.25 - Selected threshold: 0.42
- Seed: 42
- W&B run: https://wandb.ai/rl-research-team/legal-guidance-prefix-v3/runs/fru3wk9m
First-guidance-turn annotations are silver and were not independently human-validated for this run. Onset metrics must not be described as gold.
Validation results
| Evaluation | Macro-F1 | Positive F1 | AUPRC | Accuracy | Balanced accuracy |
|---|---|---|---|---|---|
| All prefixes, conversation-weighted | 0.8600 | 0.8387 | 0.8850 | 0.8632 | 0.8664 |
| All prefixes, unweighted | 0.8269 | 0.8079 | 0.8452 | 0.8289 | 0.8354 |
| Final/full prefixes | 0.8787 | 0.8699 | 0.9199 | 0.8793 | 0.8789 |
| Majority reference, weighted prefixes | 0.3740 | 0.0000 | 0.4026 | 0.5974 | 0.5000 |
Silver onset analysis
| Measure | Value |
|---|---|
| Negative-conversation false-alarm rate | 0.1474 |
| Pre-onset false-alarm rate | 0.1269 |
| Missed-guidance rate | 0.0970 |
| Exact onset accuracy | 0.7164 |
| Within-one-user-turn accuracy | 0.8209 |
| Mean absolute turn error among detected positives | 0.4215 |
Repository artifacts
tfidf_prefix_pipeline.joblib: fitted feature union, classifier, threshold, and provenanceprefix_validation_metrics.json: complete metrics, candidate results, and selected threshold curveprefix_validation_predictions.csv: per-prefix probabilities and predictionsonset_validation_predictions.csv: conversation-level onset predictionsrun_metadata.json: data, feature, software, and split provenance
Limitations
The labels and onset locations used here are silver. This is a sparse lexical
baseline and may be brittle to paraphrases, spelling changes, domain shift, and
jurisdiction-specific terminology. It has not yet been evaluated on the frozen
human gold set. Gold onset evaluation requires manual review of
first_guidance_user_turn_id.
Loading
Only load pickle/joblib files from sources you trust.
import joblib
artifact = joblib.load("tfidf_prefix_pipeline.joblib")
X = artifact["features"].transform(["User: example text"])
probability = artifact["classifier"].predict_proba(X)[:, 1]
prediction = (probability >= artifact["guidance_threshold"]).astype(int)