new upload

README.md

@@ -0,0 +1,292 @@
+# Qwen2.5-7B-Instruct — CFPB Banking Complaint Categorisation
+
+A domain-adapted large language model fine-tuned on the CFPB Consumer Complaint Database to automatically convert unstructured customer complaint narratives into structured ticket metadata for banking operations teams.
+
+---
+
+## The Problem
+
+Financial institutions process thousands of customer complaints daily across mobile apps, websites, contact centres, email, and regulatory portals. These complaints arrive as free-form text — often incomplete, ambiguous, or written by customers who do not know which banking product or issue category applies to their situation.
+
+The result is predictable: complaints get routed to the wrong team, require manual review and reassignment, and take longer to resolve than they should. Traditional classification models handle one label at a time and struggle with the nuanced language of consumer finance. A rule-based keyword system breaks down the moment a customer phrases something slightly differently.
+
+This model addresses that by treating complaint categorisation as a **structured generation task** — the model reads the complaint narrative and produces all four required ticket fields in a single inference step.
+
+---
+
+## What the Model Does
+
+Given a customer complaint narrative, the model outputs a structured JSON object containing:
+
+```json
+{
+  "product":     "Checking or savings account",
+  "sub_product": "Checking account",
+  "issue":       "Unauthorized transactions or other transaction problem",
+  "sub_issue":   "Debit card issue"
+}
+```
+
+These four fields map directly to the CFPB Consumer Complaint taxonomy and can be consumed directly by complaint management systems, business rule engines, and routing workflows — no manual classification required.
+
+### Example
+
+**Input complaint:**
+> *"I reported fraudulent transactions on my debit card and the bank reversed my provisional credit without explaining the investigation outcome. I have been trying to reach someone for three weeks and keep getting transferred."*
+
+**Model output:**
+```json
+{
+  "product":     "Checking or savings account",
+  "sub_product": "Checking account",
+  "issue":       "Unauthorized transactions or other transaction problem",
+  "sub_issue":   "Debit card issue"
+}
+```
+
+---
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| Base model | `Qwen/Qwen2.5-7B-Instruct` |
+| Fine-tuning method | LoRA (Low-Rank Adaptation) via PEFT |
+| Training hardware | AMD Instinct MI300X (192 GB VRAM) |
+| Training backend | ROCm 7.2.4 / HIP |
+| Model precision | bfloat16 |
+| Task type | Structured JSON generation (causal LM) |
+| Output format | JSON with 4 fields: product, sub_product, issue, sub_issue |
+
+---
+
+## Training Configuration
+
+### LoRA Adapter
+
+| Parameter | Value |
+|-----------|-------|
+| Rank (`r`) | 16 |
+| Alpha | 32 |
+| Dropout | 0.05 |
+| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
+| Trainable parameters | ~1% of total model parameters |
+
+The base model weights are fully frozen. Only the LoRA adapter matrices are updated during training, making this efficient both in compute and storage — the saved adapter is significantly smaller than the full model.
+
+### Training Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| Epochs | 5 (with early stopping, patience=3) |
+| Batch size per device | 8 |
+| Gradient accumulation steps | 4 |
+| Effective batch size | 32 |
+| Learning rate | 1e-4 |
+| Optimiser | AdamW (PyTorch native) |
+| LR scheduler | Linear |
+| Precision | bf16 |
+| Max sequence length | 1024 tokens |
+
+Early stopping was applied with a patience of 3 evaluation checkpoints. Prior experiments on smaller model variants showed validation loss plateauing around epoch 2–3, so early stopping prevents wasted compute without sacrificing quality.
+
+### Dataset
+
+**Source:** CFPB Consumer Complaint Database (formatted as multi-turn chat JSONL)
+
+**Splits used:**
+
+| Split | Size |
+|-------|------|
+| Train | Full dataset (no cap) |
+| Validation | 500 |
+| Test | 500 |
+
+**Sampling strategy:** Training data was sampled using proportional stratification by `product × issue` combination. This ensures that long-tail complaint categories — which would appear only once or twice in a random 500-sample draw — receive proportional representation. Without this, the model sees most issue labels fewer than 3 times, which is insufficient for reliable generation.
+
+**Chat template:** Qwen's built-in `apply_chat_template` was used to format each example into a single training string with `<|im_start|>` / `<|im_end|>` special tokens. The assistant turn (the JSON output) was included in full — no generation prompt was added at training time.
+
+---
+
+## Inference with Constrained Decoding
+
+At inference time, this model uses a **two-pass constrained decoding** approach:
+
+1. **Pass 1** — Standard greedy decoding generates the JSON output.
+2. **Pass 2** — Each field value is snapped to the nearest canonical CFPB label using TF-IDF cosine similarity (unigram + bigram features).
+
+This matters because the CFPB taxonomy contains 80+ canonical issue strings with very similar phrasing. A model that generates *"unauthorized transaction"* when the canonical label is *"unauthorized transactions or other transaction problem"* would score zero on exact match — but is semantically correct. The constrained decoder corrects these surface-level mismatches without changing the underlying prediction.
+
+Jaccard similarity was evaluated as an alternative snapping strategy but proved insufficient for near-duplicate labels (e.g., *"problem with fees"* vs *"other fee"*) where single-word differences produce high Jaccard overlap. TF-IDF on bigrams separates these reliably.
+
+---
+
+## Evaluation Results
+
+Evaluated on 500 held-out test examples from the CFPB dataset. Baseline is the unmodified `Qwen2.5-7B-Instruct` base model with no fine-tuning.
+
+### Primary Metrics — Structured JSON Extraction
+
+| Metric | Baseline | Fine-tuned | Δ |
+|--------|----------|------------|---|
+| Exact JSON Match | 0.0000 | 0.2280 | +0.2280 |
+| Avg Field Accuracy | 0.0030 | 0.5925 | +0.5895 |
+| Micro F1 | 0.0030 | 0.5925 | +0.5895 |
+| Macro F1 | 0.0008 | 0.2395 | +0.2387 |
+| Weighted F1 | 0.0059 | 0.5814 | +0.5755 |
+
+**Per-field accuracy:**
+
+| Field | Baseline | Fine-tuned | Δ |
+|-------|----------|------------|---|
+| product | 0.010 | **0.910** | +0.900 |
+| sub_product | 0.002 | **0.628** | +0.626 |
+| issue | 0.000 | **0.336** | +0.336 |
+| sub_issue | 0.000 | **0.496** | +0.496 |
+
+`product` accuracy of 91% is expected — the CFPB product taxonomy has around a dozen top-level categories and the model learns them well. `issue` at 33.6% reflects the genuine difficulty of the field: 80+ canonical strings with overlapping phrasing, many appearing infrequently even in the full training set.
+
+### Secondary Metrics — Generative Quality
+
+These metrics measure output fluency and n-gram overlap. They are secondary to the structured metrics above, but confirm the model is generating coherent, well-formed text.
+
+| Metric | Baseline | Fine-tuned | Δ |
+|--------|----------|------------|---|
+| ROUGE-1 | 0.4592 | 0.7035 | +0.2443 |
+| ROUGE-2 | 0.2523 | 0.6049 | +0.3526 |
+| ROUGE-L | 0.4258 | 0.6915 | +0.2657 |
+| BLEU | 0.0003 | 0.1905 | +0.1902 |
+| SacreBLEU | 19.77 | 65.07 | +45.30 |
+| METEOR | 0.1133 | 0.6309 | +0.5176 |
+
+The SacreBLEU jump from 19.77 to 65.07 and METEOR from 0.11 to 0.63 indicate the fine-tuned model is generating outputs that are not just structurally similar to references, but lexically aligned — which for this task means using the correct canonical CFPB terminology consistently.
+
+---
+
+## How to Use
+
+### Load the adapter
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+base_model_name = "Qwen/Qwen2.5-7B-Instruct"
+adapter_path    = "your-hf-username/qwen2.5-7b-cfpb-complaint-categorisation"
+
+tokenizer = AutoTokenizer.from_pretrained(adapter_path)
+base      = AutoModelForCausalLM.from_pretrained(
+    base_model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(base, adapter_path)
+model.eval()
+```
+
+### Run inference
+
+```python
+def categorise_complaint(complaint_text: str, model, tokenizer) -> dict:
+    messages = [
+        {
+            "role": "system",
+            "content": (
+                "You are a banking complaint classification assistant. "
+                "Given a consumer complaint narrative, extract the CFPB ticket fields "
+                "as a JSON object with keys: product, sub_product, issue, sub_issue."
+            ),
+        },
+        {
+            "role": "user",
+            "content": complaint_text,
+        },
+    ]
+
+    prompt = tokenizer.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True,
+    )
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+    with torch.no_grad():
+        output = model.generate(
+            **inputs,
+            max_new_tokens=128,
+            do_sample=False,
+            pad_token_id=tokenizer.eos_token_id,
+        )
+
+    prompt_len = inputs["input_ids"].shape[1]
+    generated  = tokenizer.decode(output[0][prompt_len:], skip_special_tokens=True)
+    return generated
+
+
+complaint = """
+I reported fraudulent transactions on my debit card and the bank reversed
+my provisional credit without explaining the investigation outcome.
+"""
+
+result = categorise_complaint(complaint, model, tokenizer)
+print(result)
+# {"product": "Checking or savings account", "sub_product": "Checking account",
+#  "issue": "Unauthorized transactions or other transaction problem",
+#  "sub_issue": "Debit card issue"}
+```
+
+---
+
+## Dependencies
+
+```
+transformers==4.44.0
+peft==0.12.0
+accelerate==0.34.0
+datasets==2.21.0
+torch (ROCm-compatible build for AMD, or standard CUDA build)
+scikit-learn
+rouge-score
+sacrebleu
+nltk
+```
+
+---
+
+## Limitations
+
+- **CFPB taxonomy only.** The model is trained on and constrained to CFPB Consumer Complaint Database labels. It is not a general-purpose complaint classifier and should not be used with complaint taxonomies from other regulatory bodies or internal systems without retraining.
+- **Issue field accuracy.** The `issue` field (33.6% accuracy) is the weakest link. The CFPB issue taxonomy contains 80+ canonical strings with overlapping phrasing. Expanding training data and further tuning the constrained decoder are the most direct paths to improvement.
+- **English language only.** All training data is in English. Performance on non-English complaints is untested and likely poor.
+- **Context length.** Complaints longer than 1024 tokens will be truncated. Most CFPB complaints are well within this limit, but very long narratives may lose relevant context.
+
+---
+
+## Intended Use
+
+This model is intended for use by:
+- Banking operations teams automating first-touch complaint categorisation
+- Compliance teams processing regulatory complaint filings
+- Contact centre platforms routing incoming complaints before agent assignment
+- Research teams studying LLM adaptation for financial NLP tasks
+
+It is not intended for consumer-facing deployment without human review of outputs, or for use in jurisdictions where automated complaint classification decisions have legal or regulatory implications without appropriate oversight.
+
+---
+
+## Training Infrastructure
+
+Trained on an AMD Instinct MI300X GPU (192 GB HBM3 VRAM) running ROCm 7.2.4. The training stack is fully ROCm-native — `bitsandbytes` (CUDA-only) is not used. Model precision is bfloat16, which is the native compute type for the CDNA3 architecture.
+
+---
+
+## Citation
+
+If you use this model in research or production, please cite the CFPB Consumer Complaint Database as the data source:
+
+```
+Consumer Financial Protection Bureau (CFPB)
+Consumer Complaint Database
+https://www.consumerfinance.gov/data-research/consumer-complaints/
+```

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-7B-Instruct",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "o_proj",
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:babd759f3bae6413a4e16efaf7d05182ca474bce50782afdb3cb6b5f8e2e900f
+size 40400200

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

baseline_metrics.json

@@ -0,0 +1,28 @@
+{
+  "field_metrics": {
+    "product": {
+      "exact_match": 0.01,
+      "precision": 0.518,
+      "recall": 0.01,
+      "f1": 0.0196
+    },
+    "sub_product": {
+      "rouge1": 0.0041,
+      "rouge2": 0.003,
+      "rougeL": 0.0041,
+      "bleu": 0.0
+    },
+    "issue": {
+      "rouge1": 0.0018,
+      "rouge2": 0.0,
+      "rougeL": 0.0018,
+      "bleu": 0.0
+    },
+    "sub_issue": {
+      "rouge1": 0.0004,
+      "rouge2": 0.0,
+      "rougeL": 0.0004,
+      "bleu": 0.0
+    }
+  }
+}

best_demo_examples.json

@@ -0,0 +1,992 @@
+[
+  {
+    "idx": 2,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nOn XXXX XXXX, 2015, my mother passed away I was named Executrix of the estate. On XXXX XXXX, 2015, I mailed a copy of the death certificate, letter of Testamentary, and letter of intent to sell the property to Wells Fargo to stop the reverse mortgage. On XXXX XXXX, 2015, the appraisal ordered by Wells Fargo was completed. On XXXX XXXX, 2015, I contacted the representative from Wells Fargo that was provided to me. The paperwork was received and I inquired as to the results of the appraisal. She stated that she had not received it and to call back in one week. On XXXX XXXX, XXXX, I again contacted Wells Fargo regarding the status of the appraisal. I was told there was no appraisal had been submitted to Wells Fargo but she would check into. She advised to use XXXX to establish a selling price. On XXXX XXXX, 2015, I left a message for the Wells Fargo and to date there has been no follow-up from Wells Fargo. On XXXX XXXX, 2015, I called Wells Fargo to obtain an email address to submit my concerns. I was advised that all communication can only be through phone or fax. On XXXX XXXX, 2015, I faxed a letter demanding a copy of the appraisal by XXXX XXXX, 2015 or I would contact your organization. To date there has been no response from Wells Fargo regarding my request.\n",
+    "ground_truth": {
+      "product": "mortgage",
+      "sub_product": "reverse mortgage",
+      "issue": "settlement process and costs",
+      "sub_issue": "not specified"
+    },
+    "base_prediction": {
+      "Product": "reverse mortgage",
+      "Sub-product": "appraisal and property management",
+      "Issue": "delayed response",
+      "Sub-issue": "failure to provide appraisal"
+    },
+    "ft_prediction": {
+      "product": "mortgage",
+      "sub_product": "reverse mortgage",
+      "issue": "settlement process and costs",
+      "sub_issue": "not specified"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 6,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nI contacted Shell Citibank XXXX XXXX. I called customer service. I asked them about a credit limit increase if it was a soft or a hard inquiry. I explained I was going to go for a major purchase and I ca n't have any hard inquiries on my credit. They said they could not answer this question. They said since you have a fraud alert to fill it out online and they will contact me before they pull the credit whether its soft or hard. This did not happen it was a hard pull the next day. I spoke to several supervisors the XXXX XXXX I spoke with said it would take up to 10 days to remove it. This did not happen. I dealt with citi executive response I spoke to a woman named XXXX XXXX she can be reached at XXXX. She did nothing at her level to resolve the issue. She said because I filled out online with my income what I was told to fill out by customer service. That I authorized my credit to be ran. She did nothing to address the negligence and incompetence of the company representative misinforming me. She does not even care I might be denied for my large purchase because of this inquiry. I think this is a very dirty tactic that they play. I have allready filed a complaint with the ftc also. I am sick and tired of companys not being held accountable by people who work for them and misinform customers. Everything was all done over the phone. My credit was pulled once through XXXX. XXXX no longer allows you to dispute through the credit bureau only with the creditor.\n",
+    "ground_truth": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "credit line increase/decrease",
+      "sub_issue": "not specified"
+    },
+    "base_prediction": {
+      "Product": "credit card",
+      "Sub-product": "credit limit",
+      "Issue": "inaccurate credit inquiry type",
+      "Sub-issue": "hard inquiry instead of soft inquiry"
+    },
+    "ft_prediction": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "credit line increase/decrease",
+      "sub_issue": "not specified"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.5623
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 9,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nIn accordance with the Fair Credit Reporting Act COMENITYBANK/COMENITY CARD Acct # XXXX XXXX  has violated my rights. \n\n15 USC 1681 Section 602 States that I have a right to privacy.\n\n15 USC 1681 Section 604 A Section 2 : It also states a consumer reporting agency can not furnish an account without my written instructions.\n\n15 USC 1666B : A creditor may not treat a payment on a credit card account under an open end consumer credit plan as late for any purpose.\n",
+    "ground_truth": {
+      "product": "credit card",
+      "sub_product": "general-purpose credit card or charge card",
+      "issue": "improper use of your report",
+      "sub_issue": "reporting company used your report improperly"
+    },
+    "base_prediction": {
+      "Product": "credit card",
+      "Sub-product": "consumer credit reporting",
+      "Issue": "violation of privacy rights",
+      "Sub-issue": "unauthorized filing of account information"
+    },
+    "ft_prediction": {
+      "product": "credit card",
+      "sub_product": "general-purpose credit card or charge card",
+      "issue": "improper use of your report",
+      "sub_issue": "reporting company used your report improperly"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 13,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nI have XXXX x J.P. Morgan Chase bank accounts. \n\nJ.P. Morgan Chase offers an 'alert service ' to their customers to reduce fraud, and in particular, to ensure that - should an account not have sufficient funds available - a customer is instantly 'alerted ' so as to be able to transfer or add funds to their account/s in time for a payment to be processed successfully. \n\nThis 'alert ' service is widely advertised as a part of the benefits of being a Chase banking or credit card customer. \n\nNote that this alert service does require some 'setting up ', which I have done, and used successfully prior to this incident. \n\nFrom XXXX XXXX 2015, the Chase alert service stopped working completely on all my accounts and Chase credit cards, and I have incurred substantial fees as a result of not receiving notifications of insufficient funds that would otherwise allow me to transfer or add funds in time to avoid such fees. Prior to this, I have used this service ritually with few issues, but on XXXX XXXX/XXXX/15, it began to fail completely, and Chase began to charge me ISF fees as a result. \n\nOnce I realised the issue, I contacted Chase Customer Service, and they insisted that their system was 'working fine ', and that I should set up alerts via 'Push ', Text and email just to be certain. I followed their advice and did so, ( receiving no less than XXXX emails confirming these changes to my alerts. However, despite setting up all alerts via all means, none of them worked at all. Meanwhile, Chase continued to charge me ISF fees, ignored my concerns raised, told me the alerts were just a 'free service ' to me, and were unwilling to assist any further. \n\nAfter several calls, emails and texts about the issue to my Chase account manager - XXXX XXXX - he finally requested I come in to demonstrate the issue to him on XXXX/XXXX/15. While in his presence, he checked my alert settings, and then had me perform XXXX x transactions. The Chase alert system failed completely for both, so he agreed it was an issue and pledged to report it to 'National ' customer service for a full refund of the fees. \n\nTo date, none of the fees have been refunded to the account that was used for demonstrating the issue, and as a result, my account/s remain in negative balance. Chase calls and writes constantly to add funds so as to bring my account/s back into credit, but I refuse to do so, as adding funds would serve only to pay Chase their ISF fees, and I am certain any request for refund would be denied. \n\nI do not work as hard as I do just to pay bank fees, especially when a failure on the banks part is the cause of such fees being incurred.\n",
+    "ground_truth": {
+      "product": "bank account or service",
+      "sub_product": "checking account",
+      "issue": "problems caused by my funds being low",
+      "sub_issue": "not specified"
+    },
+    "base_prediction": {
+      "Product": "banking",
+      "Sub-product": "checking/savings accounts",
+      "Issue": "service failure",
+      "Sub-issue": "alert service not working"
+    },
+    "ft_prediction": {
+      "product": "bank account or service",
+      "sub_product": "checking account",
+      "issue": "problems caused by my funds being low",
+      "sub_issue": "not specified"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 15,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nXXXX issues : First, I opened a capital one credit card and had 3 months to spend {$XXXX}. On a recorded line on XXXX different occasions with XXXX different agents, they verified that my spending date was until XX/XX/XXXX. I spent over {$XXXX} by that date. Once the date had passed, I did not see the XXXX miles that I was owed. I called and they said that I needed to spend it by XX/XX/XXXX. My final payment, putting me over my limit went through on XX/XX/XXXX. I have asked for them to review the conversations on recorded lines in which it was clearly stated that I had until the XXXX, otherwise, I would have made my final purchase prior to the date. It has been over 2 months since that call was placed and I was told weeks ago that there would be follow up, which there has not been. \n\nAdditionally, the promotion states that it is {$XXXX} to open a card and you receive {$XXXX} with of travel credit in my account. I have not seen that either. WHen I called, I was then told that I will only receive the XXXX after I renew the card the following year for another {$XXXX}. I have spent at least XXXX hours on recorded lines with multiple supervisors and have gotten no where. I am, at this point, going to start seeking legal action for the sheer ethos of it alone.\n",
+    "ground_truth": {
+      "product": "credit card",
+      "sub_product": "general-purpose credit card or charge card",
+      "issue": "advertising and marketing, including promotional offers",
+      "sub_issue": "didn't receive advertised or promotional terms"
+    },
+    "base_prediction": {
+      "Product": "credit card",
+      "Sub-product": "travel rewards credit card",
+      "Issue": "promotion not honored",
+      "Sub-issue": "unfulfilled promised rewards"
+    },
+    "ft_prediction": {
+      "product": "credit card",
+      "sub_product": "general-purpose credit card or charge card",
+      "issue": "advertising and marketing, including promotional offers",
+      "sub_issue": "didn't receive advertised or promotional terms"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 16,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nI am submitting this complaint regarding American Express and its handling of my Delta American Express credit card benefits after closing all of my accounts. American Express placed all of my accounts on hold and then closed them, despite my being in good standing for more than 20 years. As a result, I no longer have an active American Express card. However, I had already paid the annual fee for my Delta American Express card, whose primary benefit is the annual Delta Companion Certificate.\n\nThe Delta Companion Certificate is currently visible and valid in my Delta\nSkyMiles account and is set to expire on XX/XX/year>. Delta has confirmed that in order to redeem the certificate, the required taxes and fees must be paid using an American Express card. Because American Express closed all of my accounts, I am unable to use the certificate. Delta has advised that it can not override this American Express requirement.\n\nAs a result, I paid an annual fee for a benefit that I am now unable to use through no fault of my own. American Express has effectively denied access to a paid card benefit while retaining the annual fee. This has caused me financial harm and loss of value. I have also experienced related losses with other co-branded programs, including Hilton Honors, due to the account closures.\n",
+    "ground_truth": {
+      "product": "credit card",
+      "sub_product": "general-purpose credit card or charge card",
+      "issue": "closing your account",
+      "sub_issue": "company closed your account"
+    },
+    "base_prediction": {
+      "Product": "credit card",
+      "Sub-product": "co-branded credit card",
+      "Issue": "account closure",
+      "Sub-issue": "denied access to paid benefits after account closure"
+    },
+    "ft_prediction": {
+      "product": "credit card",
+      "sub_product": "general-purpose credit card or charge card",
+      "issue": "closing your account",
+      "sub_issue": "company closed your account"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.5623
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 21,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nIn XXXX of 2016 I purchased a XXXX from XXXX that used to be on display and was marked down. I purchased it from them for around {$600.00}. After about a month I returned the item to the store and received a return receipt. In XXXX XXXX I was reviewing my account when I noticed that the charge for the XXXX had yet to be credited to my account. I contacted Citi Retail Services and Disputed the Charge. On XXXX XXXX I received a letter from them ( Enclosed ) stating they were going to award me credit for the charge in the amount of {$570.00}, and in the mail the week after I received the award letter I received an updated account statement showing the credit had been removed. My account balance at the time was somewhere around {$1400.00}. After waiting for 2 weeks my account balance had not changed. I contacted Customer Service and spoke with an associate. They stated they had already credited me the balance stated in the letter, but could not remove it from my current balance because they had already updated the statement from XXXX. This confused me mainly because when someone is given credit it typically means they have the amount deducted from their accounts, not just have a previous statement edited.\n",
+    "ground_truth": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "billing disputes",
+      "sub_issue": "not specified"
+    },
+    "base_prediction": {
+      "Product": "credit card",
+      "Sub-product": "retail card",
+      "Issue": "incorrect account balance",
+      "Sub-issue": "delayed credit application"
+    },
+    "ft_prediction": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "billing disputes",
+      "sub_issue": "not specified"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 26,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nName of Financial Institution against whom I am filing this complaint : Regions Bank, XXXX XXXX XXXX, XXXX XXXX, FL XXXX Names of Regions Bank employees against whom I am filing this complaint : XXXX XXXX, Mortgage Loan Originator, Vice President ( M : XXXX ; W : XXXX, XXXXXXXXXXXX ; XXXX XXXX XXXX, XXXX XXXX XXXX, Regions Mortgage ( W : XXXX, M : XXXX, XXXXXXXXXXXX ) On XXXX XXXX, 2016, I requested XXXX XXXX XXXX refund my credit card the {$420.00} for the Regions appraisal fee that was PREMATURELY ordered by Regions Bank XXXX Prior to the appraisal, XXXX XXXX assured me that the appraisal for a property I was pursuing would not be performed UNTIL the home inspection was performed and any necessary repairs resolved by contract between me and the seller. I also informed XXXX XXXX that I would not be moving forward with Regions Bank for my mortgage financing. \nNearly ( 6 ) weeks have passed with very little or no communication from XXXX XXXX regarding my request for reimbursement, even though via email XXXX XXXX stated it was being processed and that he would check on the status of my request. Furthermore, ( 2 ) weeks ago I escalated my request to XXXX XXXX 's boss XXXX XXXX, but with no resolution, communication or reimbursement to date. \nTherefore, as a next step to get this matter resolved, and my money reimbursed, I am submitting my complaint to the Consumer Financial Protection Bureau. \nThank you for your assistance!\n",
+    "ground_truth": {
+      "product": "mortgage",
+      "sub_product": "conventional fixed mortgage",
+      "issue": "application, originator, mortgage broker",
+      "sub_issue": "not specified"
+    },
+    "base_prediction": {
+      "Product": "credit cards",
+      "Sub-product": "appraisal fees",
+      "Issue": "refund request",
+      "Sub-issue": "delayed response and reimbursement"
+    },
+    "ft_prediction": {
+      "product": "mortgage",
+      "sub_product": "conventional fixed mortgage",
+      "issue": "application, originator, mortgage broker",
+      "sub_issue": "not specified"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.5623
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 1.0
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 28,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nXXXX use my XXXX ( provided by Wells Fargo bank ) card number ( ending in XXXX exp XXXX ) for recurring charges. \nSince card ending in XXXX expires on XXXX Wells Fargo card company send another credit card ending in XXXX exp XXXX. \n\nBut XXXX has old card number ( ending XXXX and exp XXXX ) on their file. I never updated billing card information in XXXX. Neither I activated new card ( ending XXXX ). \nBut still Wells fargo accepted credit card charge from expired card. \n\nI was sure that since the card has expired XXXX will not be able to charge using old card information. But Wells Fargo card service has accepted the charge on XXXX and XXXX. Since I did not use this card anywhere I did not pay attention to Wells Fargo card account. Hence I miss payment. On XXXX I got billing statement and found that I owe {$91.00} with the late fee of {$27.00}. \nI contacted company customer care and they inform that since my account number remain same they will accept the charges from expired credit card. But when I check my new statement account number it ends in XXXX. \nMy question is will credit card company accept the charges on expired credit card.\n",
+    "ground_truth": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "billing statement",
+      "sub_issue": "not specified"
+    },
+    "base_prediction": {
+      "Product": "credit card",
+      "Sub-product": "payment processing",
+      "Issue": "unauthorized charges",
+      "Sub-issue": "charges on expired card"
+    },
+    "ft_prediction": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "billing statement",
+      "sub_issue": "not specified"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  },
+  {
+    "idx": 29,
+    "complaint": "\nAnalyze the following customer complaint and determine the appropriate complaint classification.\n\nCustomer Complaint:\nChase credit card withheld information about the cut off time XXXX east time for payments on XXXX/XXXX/2015. Then at XXXX I called to pay it to avoid late to pay with chase checking the said even though I paid in past with that account I would have provide my account and routing number with I did n't have on me. I spoke to supervisor who said on a recorded line not to worry he will note on the account that he would refund any late fee was I paid it on the XXXX/XXXX/2015. However now chase still charged me a late fee. Even though I paid on the XXXX. In conclusion CHASE credit card automated machine didnot disclose that there was a cut off time on the XXXX due date. And they are scamming me and other customers out of XXXX of dollars. They need to be held accountable for ther scheme\n",
+    "ground_truth": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "late fee",
+      "sub_issue": "not specified"
+    },
+    "base_prediction": {
+      "Product": "credit card",
+      "Sub-product": "payment processing",
+      "Issue": "late fee assessment",
+      "Sub-issue": "incorrect due date information"
+    },
+    "ft_prediction": {
+      "product": "credit card",
+      "sub_product": "not specified",
+      "issue": "late fee",
+      "sub_issue": "not specified"
+    },
+    "base_metrics": {
+      "product": {
+        "exact_match": 0.0,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0
+      },
+      "sub_product": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      },
+      "sub_issue": {
+        "rouge1": 0.0,
+        "rouge2": 0.0,
+        "rougeL": 0.0,
+        "bleu": 0
+      }
+    },
+    "ft_metrics": {
+      "product": {
+        "exact_match": 1.0,
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1": 1.0
+      },
+      "sub_product": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      },
+      "sub_issue": {
+        "rouge1": 1.0,
+        "rouge2": 1.0,
+        "rougeL": 1.0,
+        "bleu": 0.3162
+      }
+    },
+    "improvement": {
+      "product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_product": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "sub_issue": {
+        "base_correct": 0,
+        "ft_correct": 1,
+        "improved": 1
+      },
+      "base_total_correct": 0,
+      "ft_total_correct": 4,
+      "total_improvement": 4
+    }
+  }
+]

final_metrics.json

@@ -0,0 +1,28 @@
+{
+  "field_metrics": {
+    "product": {
+      "exact_match": 0.908,
+      "precision": 0.9082,
+      "recall": 0.908,
+      "f1": 0.9068
+    },
+    "sub_product": {
+      "rouge1": 0.7122,
+      "rouge2": 0.6452,
+      "rougeL": 0.7122,
+      "bleu": 0.5026
+    },
+    "issue": {
+      "rouge1": 0.4018,
+      "rouge2": 0.3463,
+      "rougeL": 0.4013,
+      "bleu": 0.3368
+    },
+    "sub_issue": {
+      "rouge1": 0.5215,
+      "rouge2": 0.4895,
+      "rougeL": 0.5207,
+      "bleu": 0.2283
+    }
+  }
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}