OpenLEWS-14B-v1

OpenLEWS-14B-v1 is a LoRA adapter for Qwen/Qwen2.5-14B that forecasts whether a drug, medical device, or exposure will be consolidated into a federal MDL within roughly 18 months of an evidence-cutoff date, given an as-of-date evidence dossier. It reads a dossier, writes a step-by-step forecasting rationale, and ends with a JSON probability.

It accompanies the paper Signals Beat Scale: Evidence Acquisition Dominates Model Choice in Forecasting Mass-Tort Consolidation (Lee & Tandon, Decover AI; arXiv link forthcoming) and is the 14B model evaluated there on the LEWS 1.0 benchmark. Decover AI deploys the surrounding system in production as the Litigation Early Warning System (LEWS). The benchmark itself (all 167 as-of-date dossiers, labels, and the evaluation harness) is released at leejason2026/lews-bench.

Training data and provenance

• Training targets are step-by-step forecasting rationales, each ending in a JSON probability, generated by Claude Sonnet 4.6 over training dossiers, following the rationale-distillation recipe of Hsieh et al. (2023).
• Outcome labels derive from public JPML records. The 167 evaluation cases comprise 78 granted petitions, 19 denied petitions, 51 pre-petition snapshots of the granted substances (36 months earlier, labeled negative), and 19 never-consolidated substances; see the benchmark repo for the full composition.
• Evidence dossiers are reconstructed strictly as of each petition date from a ~17M-item signal corpus (FAERS/MAUDE, PubMed, federal dockets, FDA actions, news, public forums, plus leading indicators such as plaintiff-firm intake advertising, litigation-funding activity, terminated clinical trials, and search interest). The live signal corpus and the acquisition pipeline that produces current dossiers are a commercial asset of LEWS and are not released; the 167 frozen benchmark dossiers are released in the benchmark repo.

This released adapter was trained on all 167 examples. The evaluation numbers below come from the paper's held-out protocols (5-fold out-of-fold CV and a strict temporal split), in which no model predicts a petition it saw in training.

Evaluation (LEWS 1.0, n=167)

Model	AUROC (5-fold OOF)
Untrained heuristic	0.691
Logistic regression (21 features)	0.829
OpenLEWS-7B	0.856 ± 0.011 (4 runs)
HistGBM (21 features)	0.871
OpenLEWS-14B (this model class)	0.869 ± 0.016 (3 runs)
Claude Sonnet 4.6 (API)	0.903

• Across the three training runs the AUROC replicates were 0.884, 0.847, and 0.875 (all 167/167 coverage); the gap to Claude Sonnet 4.6 reaches conventional significance in one of the three runs (DeLong p = 0.011, 0.149, 0.350). See the paper for per-run statistics.
• On the hard subset of real petitions only (grant vs. deny, n=97) every model in the study drops sharply (frontier model 0.661; this model class retains modest skill); full-benchmark numbers describe the screening regime. See the paper's hard-subset analysis.
• Generation coverage in evaluation: 167/167 dossiers produced a parseable probability. Treat a non-answer as a non-answer; do not impute 0.5.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen2.5-14B"
tok = AutoTokenizer.from_pretrained("decoverai/OpenLEWS-14B-v1")
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "decoverai/OpenLEWS-14B-v1")

INSTR = (
    "\n\nForecast whether this will consolidate into a U.S. federal MDL within ~18 months, "
    "using ONLY the information above. Reason step by step, then end with a JSON line "
    '{"probability": <0..1>, "tier": "HIGH|MEDIUM|LOW"}.\n\nANALYSIS:\n'
)

dossier = "..."  # your as-of-date evidence dossier (see paper, Section 3)
ids = tok(dossier + INSTR, return_tensors="pt", truncation=True, max_length=8192).to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

Parse the last {"probability": ...} JSON object in the output. If none is present, the model did not answer; do not default to 0.5.

Intended use and limitations

• Decision support for prioritizing legal-research investigations. Not legal advice, and not a substitute for professional judgment; predictions concern petition outcomes, not the merits of any claim.
• Trained and evaluated on n=167 petitions; run-to-run variance is material (±0.016 AUROC across training runs). Confidence intervals in the paper are wide.
• Input must follow the dossier + instruction format above; performance on free-form prompts is untested. English, U.S. federal MDL practice only.
• Output probabilities inherit the calibration reported in the paper; recalibrate before treating them as frequencies.

Citation

@article{lee2026signals,
  title  = {Signals Beat Scale: Evidence Acquisition Dominates Model Choice in Forecasting Mass-Tort Consolidation},
  author = {Lee, Jason and Tandon, Ravi},
  year   = {2026},
  note   = {arXiv preprint, forthcoming}
}