aryachakraborty
/

arya-cfpb-qwen25-7b-lora

@@ -1,202 +1,292 @@
----
-base_model: Qwen/Qwen2.5-7B-Instruct
-library_name: peft
----
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
-### Framework versions
-- PEFT 0.12.0

+# Qwen2.5-7B-Instruct — CFPB Banking Complaint Categorisation
+A domain-adapted large language model fine-tuned on the CFPB Consumer Complaint Database to automatically convert unstructured customer complaint narratives into structured ticket metadata for banking operations teams.
+---
+## The Problem
+Financial institutions process thousands of customer complaints daily across mobile apps, websites, contact centres, email, and regulatory portals. These complaints arrive as free-form text — often incomplete, ambiguous, or written by customers who do not know which banking product or issue category applies to their situation.
+The result is predictable: complaints get routed to the wrong team, require manual review and reassignment, and take longer to resolve than they should. Traditional classification models handle one label at a time and struggle with the nuanced language of consumer finance. A rule-based keyword system breaks down the moment a customer phrases something slightly differently.
+This model addresses that by treating complaint categorisation as a **structured generation task** — the model reads the complaint narrative and produces all four required ticket fields in a single inference step.
+---
+## What the Model Does
+Given a customer complaint narrative, the model outputs a structured JSON object containing:
+```json
+{
+  "product":     "Checking or savings account",
+  "sub_product": "Checking account",
+  "issue":       "Unauthorized transactions or other transaction problem",
+  "sub_issue":   "Debit card issue"
+}
+```
+These four fields map directly to the CFPB Consumer Complaint taxonomy and can be consumed directly by complaint management systems, business rule engines, and routing workflows — no manual classification required.
+### Example
+**Input complaint:**
+> *"I reported fraudulent transactions on my debit card and the bank reversed my provisional credit without explaining the investigation outcome. I have been trying to reach someone for three weeks and keep getting transferred."*
+**Model output:**
+```json
+{
+  "product":     "Checking or savings account",
+  "sub_product": "Checking account",
+  "issue":       "Unauthorized transactions or other transaction problem",
+  "sub_issue":   "Debit card issue"
+}
+```
+---
+## Model Details
+| Property | Value |
+|----------|-------|
+| Base model | `Qwen/Qwen2.5-7B-Instruct` |
+| Fine-tuning method | LoRA (Low-Rank Adaptation) via PEFT |
+| Training hardware | AMD Instinct MI300X (192 GB VRAM) |
+| Training backend | ROCm 7.2.4 / HIP |
+| Model precision | bfloat16 |
+| Task type | Structured JSON generation (causal LM) |
+| Output format | JSON with 4 fields: product, sub_product, issue, sub_issue |
+---
+## Training Configuration
+### LoRA Adapter
+| Parameter | Value |
+|-----------|-------|
+| Rank (`r`) | 16 |
+| Alpha | 32 |
+| Dropout | 0.05 |
+| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
+| Trainable parameters | ~1% of total model parameters |
+The base model weights are fully frozen. Only the LoRA adapter matrices are updated during training, making this efficient both in compute and storage — the saved adapter is significantly smaller than the full model.
+### Training Hyperparameters
+| Parameter | Value |
+|-----------|-------|
+| Epochs | 5 (with early stopping, patience=3) |
+| Batch size per device | 8 |
+| Gradient accumulation steps | 4 |
+| Effective batch size | 32 |
+| Learning rate | 1e-4 |
+| Optimiser | AdamW (PyTorch native) |
+| LR scheduler | Linear |
+| Precision | bf16 |
+| Max sequence length | 1024 tokens |
+Early stopping was applied with a patience of 3 evaluation checkpoints. Prior experiments on smaller model variants showed validation loss plateauing around epoch 2–3, so early stopping prevents wasted compute without sacrificing quality.
+### Dataset
+**Source:** CFPB Consumer Complaint Database (formatted as multi-turn chat JSONL)
+**Splits used:**
+| Split | Size |
+|-------|------|
+| Train | Full dataset (no cap) |
+| Validation | 500 |
+| Test | 500 |
+**Sampling strategy:** Training data was sampled using proportional stratification by `product × issue` combination. This ensures that long-tail complaint categories — which would appear only once or twice in a random 500-sample draw — receive proportional representation. Without this, the model sees most issue labels fewer than 3 times, which is insufficient for reliable generation.
+**Chat template:** Qwen's built-in `apply_chat_template` was used to format each example into a single training string with `<|im_start|>` / `<|im_end|>` special tokens. The assistant turn (the JSON output) was included in full — no generation prompt was added at training time.
+---
+## Inference with Constrained Decoding
+At inference time, this model uses a **two-pass constrained decoding** approach:
+1. **Pass 1** — Standard greedy decoding generates the JSON output.
+2. **Pass 2** — Each field value is snapped to the nearest canonical CFPB label using TF-IDF cosine similarity (unigram + bigram features).
+This matters because the CFPB taxonomy contains 80+ canonical issue strings with very similar phrasing. A model that generates *"unauthorized transaction"* when the canonical label is *"unauthorized transactions or other transaction problem"* would score zero on exact match — but is semantically correct. The constrained decoder corrects these surface-level mismatches without changing the underlying prediction.
+Jaccard similarity was evaluated as an alternative snapping strategy but proved insufficient for near-duplicate labels (e.g., *"problem with fees"* vs *"other fee"*) where single-word differences produce high Jaccard overlap. TF-IDF on bigrams separates these reliably.
+---
+## Evaluation Results
+Evaluated on 500 held-out test examples from the CFPB dataset. Baseline is the unmodified `Qwen2.5-7B-Instruct` base model with no fine-tuning.
+### Primary Metrics — Structured JSON Extraction
+| Metric | Baseline | Fine-tuned | Δ |
+|--------|----------|------------|---|
+| Exact JSON Match | 0.0000 | 0.2280 | +0.2280 |
+| Avg Field Accuracy | 0.0030 | 0.5925 | +0.5895 |
+| Micro F1 | 0.0030 | 0.5925 | +0.5895 |
+| Macro F1 | 0.0008 | 0.2395 | +0.2387 |
+| Weighted F1 | 0.0059 | 0.5814 | +0.5755 |
+**Per-field accuracy:**
+| Field | Baseline | Fine-tuned | Δ |
+|-------|----------|------------|---|
+| product | 0.010 | **0.910** | +0.900 |
+| sub_product | 0.002 | **0.628** | +0.626 |
+| issue | 0.000 | **0.336** | +0.336 |
+| sub_issue | 0.000 | **0.496** | +0.496 |
+`product` accuracy of 91% is expected — the CFPB product taxonomy has around a dozen top-level categories and the model learns them well. `issue` at 33.6% reflects the genuine difficulty of the field: 80+ canonical strings with overlapping phrasing, many appearing infrequently even in the full training set.
+### Secondary Metrics — Generative Quality
+These metrics measure output fluency and n-gram overlap. They are secondary to the structured metrics above, but confirm the model is generating coherent, well-formed text.
+| Metric | Baseline | Fine-tuned | Δ |
+|--------|----------|------------|---|
+| ROUGE-1 | 0.4592 | 0.7035 | +0.2443 |
+| ROUGE-2 | 0.2523 | 0.6049 | +0.3526 |
+| ROUGE-L | 0.4258 | 0.6915 | +0.2657 |
+| BLEU | 0.0003 | 0.1905 | +0.1902 |
+| SacreBLEU | 19.77 | 65.07 | +45.30 |
+| METEOR | 0.1133 | 0.6309 | +0.5176 |
+The SacreBLEU jump from 19.77 to 65.07 and METEOR from 0.11 to 0.63 indicate the fine-tuned model is generating outputs that are not just structurally similar to references, but lexically aligned — which for this task means using the correct canonical CFPB terminology consistently.
+---
+## How to Use
+### Load the adapter
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+base_model_name = "Qwen/Qwen2.5-7B-Instruct"
+adapter_path    = "your-hf-username/qwen2.5-7b-cfpb-complaint-categorisation"
+tokenizer = AutoTokenizer.from_pretrained(adapter_path)
+base      = AutoModelForCausalLM.from_pretrained(
+    base_model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(base, adapter_path)
+model.eval()
+```
+### Run inference
+```python
+def categorise_complaint(complaint_text: str, model, tokenizer) -> dict:
+    messages = [
+        {
+            "role": "system",
+            "content": (
+                "You are a banking complaint classification assistant. "
+                "Given a consumer complaint narrative, extract the CFPB ticket fields "
+                "as a JSON object with keys: product, sub_product, issue, sub_issue."
+            ),
+        },
+        {
+            "role": "user",
+            "content": complaint_text,
+        },
+    ]
+    prompt = tokenizer.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True,
+    )
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    with torch.no_grad():
+        output = model.generate(
+            **inputs,
+            max_new_tokens=128,
+            do_sample=False,
+            pad_token_id=tokenizer.eos_token_id,
+        )
+    prompt_len = inputs["input_ids"].shape[1]
+    generated  = tokenizer.decode(output[0][prompt_len:], skip_special_tokens=True)
+    return generated
+complaint = """
+I reported fraudulent transactions on my debit card and the bank reversed
+my provisional credit without explaining the investigation outcome.
+"""
+result = categorise_complaint(complaint, model, tokenizer)
+print(result)
+# {"product": "Checking or savings account", "sub_product": "Checking account",
+#  "issue": "Unauthorized transactions or other transaction problem",
+#  "sub_issue": "Debit card issue"}
+```
+---
+## Dependencies
+```
+transformers==4.44.0
+peft==0.12.0
+accelerate==0.34.0
+datasets==2.21.0
+torch (ROCm-compatible build for AMD, or standard CUDA build)
+scikit-learn
+rouge-score
+sacrebleu
+nltk
+```
+---
+## Limitations
+- **CFPB taxonomy only.** The model is trained on and constrained to CFPB Consumer Complaint Database labels. It is not a general-purpose complaint classifier and should not be used with complaint taxonomies from other regulatory bodies or internal systems without retraining.
+- **Issue field accuracy.** The `issue` field (33.6% accuracy) is the weakest link. The CFPB issue taxonomy contains 80+ canonical strings with overlapping phrasing. Expanding training data and further tuning the constrained decoder are the most direct paths to improvement.
+- **English language only.** All training data is in English. Performance on non-English complaints is untested and likely poor.
+- **Context length.** Complaints longer than 1024 tokens will be truncated. Most CFPB complaints are well within this limit, but very long narratives may lose relevant context.
+---
+## Intended Use
+This model is intended for use by:
+- Banking operations teams automating first-touch complaint categorisation
+- Compliance teams processing regulatory complaint filings
+- Contact centre platforms routing incoming complaints before agent assignment
+- Research teams studying LLM adaptation for financial NLP tasks
+It is not intended for consumer-facing deployment without human review of outputs, or for use in jurisdictions where automated complaint classification decisions have legal or regulatory implications without appropriate oversight.
+---
+## Training Infrastructure
+Trained on an AMD Instinct MI300X GPU (192 GB HBM3 VRAM) running ROCm 7.2.4. The training stack is fully ROCm-native — `bitsandbytes` (CUDA-only) is not used. Model precision is bfloat16, which is the native compute type for the CDNA3 architecture.
+---
+## Citation
+If you use this model in research or production, please cite the CFPB Consumer Complaint Database as the data source:
+```
+Consumer Financial Protection Bureau (CFPB)
+Consumer Complaint Database
+https://www.consumerfinance.gov/data-research/consumer-complaints/
+```