---
library_name: peft
base_model: microsoft/deberta-v3-small
tags:
  - text-classification
  - legal-nlp
  - contract-review
  - lora
  - ledgar
datasets:
  - coastalcph/lex_glue
metrics:
  - accuracy
  - f1
---

# Contract Clause Risk Classifier (LoRA)

A LoRA-fine-tuned `microsoft/deberta-v3-small` that classifies contract
clauses into their legal category, as part of an end-to-end contract risk
review pipeline (fine-tuning → RAG over precedent clauses → agentic
explanation generation → confidence-based human review routing).

## Model Details

- **Base model:** `microsoft/deberta-v3-small`
- **Fine-tuning method:** LoRA (`r=8`, `alpha=16`, `dropout=0.1`, targeting
  `query_proj`, `key_proj`, `value_proj`)
- **Task:** Multi-class clause classification, 100 categories
- **Training data:** [LEDGAR](https://huggingface.co/datasets/coastalcph/lex_glue)
  (`lex_glue`, `ledgar` config) — 60,000 train / 10,000 validation / 10,000
  test examples, real clause text drawn from SEC contract filings
- **Training regime:** 3 epochs, learning rate 3e-4, batch size 16
- **Hardware:** single T4 GPU (Colab)

## Results (test set, n=10,000)

| Metric | Value |
|---|---|
| Accuracy | 0.797 |
| Macro F1 | 0.643 |
| Eval loss | 0.784 |
| Throughput | ~84.5 samples/sec |

**Read macro-F1 alongside accuracy, not instead of it.** Accuracy is pulled
up by a handful of common, easy categories; macro-F1 weights every category
equally and exposes that performance is uneven across the full label set.

### Per-category performance

Strongest categories (F1 > 0.96):

| Category | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| Financial Statements | 0.988 | 0.988 | 0.988 | 82 |
| Counterparts | 0.970 | 0.998 | 0.984 | 490 |
| Use Of Proceeds | 0.975 | 0.975 | 0.975 | 120 |
| Base Salary | 0.982 | 0.964 | 0.973 | 112 |
| Waiver Of Jury Trials | 0.964 | 0.973 | 0.969 | 111 |

Weakest categories (F1 = 0.0 — model never correctly predicted these on the
test set):

| Category | Support |
|---|---|
| Applicable Laws | 53 |
| Defined Terms | 56 |
| Indemnity | 29 |
| Modifications | 55 |
| Assigns | 4 |
| Costs | 15 |
| Books | 2 |
| Qualifications | 5 |

**25 of 100 categories fall below F1 = 0.5.** The near-zero-support
categories (Books, n=2; Assigns, n=4) are unsurprising — there's barely any
signal to learn from. But `Indemnity` (n=29) and `Applicable Laws` (n=53) and
`Defined Terms` (n=56) failing completely despite reasonable support is a
real weakness, not a data-starvation artifact, and is exactly the kind of
category a fairness/bias guardrail should catch before this model is trusted
unsupervised on categories that matter for actual risk decisions.

### Inference latency

Measured on 50 samples, single T4 GPU, classifier forward pass only (does
not include RAG retrieval or LLM summary generation from the full pipeline):

| | Value |
|---|---|
| Mean | 25.97 ms |
| p50 | 26.44 ms |
| p95 | 35.02 ms |

### Estimated time savings

Based on an **assumed** 3 minutes of manual review per clause (adjust to
your own team's real average — this is a placeholder, not measured data):

| Clauses | Manual review | Automated | Time saved |
|---|---|---|---|
| 100 | 5.0 hrs | ~4 sec | ~5.0 hrs |
| 1,000 | 50.0 hrs | ~7 sec | ~50.0 hrs |
| 10,000 | 500.0 hrs | ~72 sec | ~499.9 hrs |

These numbers cover classification only. They do not include the LLM
summary-generation step, which is the slower part of the full pipeline in
practice.

## Known limitation: risk-tier mapping needs correction

This model outputs a clause **category** (e.g. `Indemnity`, `Governing
Laws`). A separate rule-based lookup maps categories to a risk tier
(High/Medium/Low) for the downstream pipeline. As shipped, **only 3 of 11
intended high-risk category names actually match LEDGAR's real label
strings** (e.g. the dataset uses `Indemnity`, not `Indemnification`), so the
risk-tier output should not be trusted until that mapping is corrected
against the model's actual `id2label` values. The classifier's category
predictions above are accurate to what's reported; the risk-tier labeling
built on top of them currently is not.

## Intended Use

Prototype / portfolio component for automating a first pass of contract
clause categorization ahead of legal review. **Not** a substitute for legal
review — low-confidence and weak-category predictions should be routed to a
human, which is what the confidence-threshold guardrail in the full pipeline
does.

## Bias, Risks, and Limitations

- Training labels come from LEDGAR (real SEC filing clauses), which is a
  reasonable proxy for clause type but was not curated by legal experts for
  this specific risk-review use case.
- Performance is uneven across categories (see per-category table above);
  25/100 categories underperform, several with zero correct predictions on
  held-out data.
- Latency figures reflect classification only, not the full RAG + LLM
  pipeline.
- Risk-tier labels are currently unreliable — see limitation above — and
  should not be used until the category-name mapping is fixed.

## Training Procedure

```python
lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    target_modules=["query_proj", "key_proj", "value_proj"],
)
```

3 epochs, learning rate `3e-4`, batch size 16, evaluated each epoch on the
validation split, best checkpoint selected by macro-F1.

## How to Use

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

base = AutoModelForSequenceClassification.from_pretrained(
    "microsoft/deberta-v3-small", num_labels=100
)
model = PeftModel.from_pretrained(base, "ashrafksalim/contract-clause-risk-lora")
tokenizer = AutoTokenizer.from_pretrained("ashrafksalim/contract-clause-risk-lora")

inputs = tokenizer("Each party shall indemnify and hold harmless...", return_tensors="pt")
logits = model(**inputs).logits
```