# Financial Question Answering with BART + LoRA Fine-Tuning ## 1. Introduction This project explores the task of financial question answering, specifically focusing on generating answers based on provided context snippets, often simulating Retrieval-Augmented Generation (RAG) scenarios. Accurate information extraction is crucial in finance, but LLMs can sometimes struggle with domain-specific factuality. To address this, we fine-tuned the `facebook/bart-large-cnn` model using Low-Rank Adaptation (LoRA) on a financial QA dataset derived from financial_phrasebank. The goal was to improve the model's ability to generate concise, contextually grounded answers. Post-training benchmarks using an LLM-as-Judge approach (Gemini) and attempted evaluations with RAG-specific frameworks (like Ragas) indicated **mixed results**. While fine-tuning potentially **improved the model's faithfulness** (generating answers more consistent with the provided context), it **did not show clear improvements in answer relevancy or overall quality scores** compared to the baseline model, and may have slightly decreased performance in these areas for some examples. Automated RAG benchmarks further highlighted challenges with context retrieval precision in the experimental setup. ## 2. Training Data The dataset used for fine-tuning are financial_phrasebank. Texts were loaded using LangChain's `UnstructuredURLLoader` and chunked using `RecursiveCharacterTextSplitter`. Question-Answer pairs were constructed based on these chunks. ## 3. Training Method The chosen training method was LoRA (Low-Rank Adaptation), applied to the `facebook/bart-large-cnn` base model. LoRA allows efficient adaptation of large models using low-rank updates while keeping most base model parameters frozen. The LoRA configuration used was: * `r=8` * `lora_alpha=16` * `lora_dropout=0.1` * `task_type=SEQ_2_SEQ_LM` * `target_modules=["q_proj", "v_proj"]` (Note: target modules depend on the base model architecture) Training Arguments (`Seq2SeqTrainingArguments`): * Learning rate: `2e-4` * Epochs: `2` (Adjusted based on training logs provided) * Batch size: `2` per device * `eval_strategy`: `"epoch"` * `save_strategy`: `"epoch"` * `load_best_model_at_end`: `True` * `metric_for_best_model`: `"eval_loss"` * Other standard arguments (weight decay, save limits, logging, etc.) The model was trained using the `Seq2SeqTrainer` from the Hugging Face `transformers` library. ## 4. Evaluation Post-training evaluation focused on assessing the quality of answers generated by the fine-tuned model compared to the baseline `facebook/bart-large-cnn` model within a simulated RAG context (using `financial_phrasebank` as the context source for benchmarking). **Evaluation Methods:** 1. **LLM-as-Judge (Gemini):** * A capable LLM (Gemini-1.5-Flash) was used as an impartial judge to evaluate the quality of answers generated by both the baseline and the fine-tuned models for the same set of questions and retrieved contexts. * Criteria included: Answer Relevancy, Faithfulness (presumed consistency with context), Fluency, Helpfulness (relative to context), and Overall Quality. * **Findings:** The fine-tuned model showed potential improvements in **faithfulness** (less likely to generate answers inconsistent with the context). Helpfulness scores saw minor increases in some cases. However, **answer relevancy and overall quality scores did not show clear improvement** and potentially decreased slightly compared to the baseline. Fluency remained high for both. 2. **Automated RAG Benchmarks (Ragas/DeepEval - Attempted):** * Frameworks like Ragas and DeepEval were used to attempt calculation of metrics such as `faithfulness`, `answer_relevancy`, and `context_precision`. * **Findings & Caveats:** Initial runs suggested similar trends to the LLM-as-Judge (potential decrease in faithfulness/relevancy post-tuning). A key finding was consistently low **context precision**, indicating significant challenges with the retriever component in finding relevant context snippets from the `financial_phrasebank` for the questions asked. **However, these benchmark runs were incomplete due to evaluation setup issues (missing API keys for default metric models), resulting in many NaN/failed calculations. Therefore, quantitative scores from these runs are unreliable and not reported here.** **Summary:** Fine-tuning with LoRA improved the model's tendency to stick to the provided context (faithfulness) but did not demonstrably improve, and may have slightly hindered, its ability to generate the most relevant or highest overall quality answers for this specific task and evaluation setup compared to the baseline BART model. Retrieval quality (context precision) was a significant challenge throughout. ## 5. Usage and Intended Uses The model is intended for use in applications requiring question answering based on provided financial text snippets, such as assisting analysts, summarizing news, or powering chatbots *where the context is explicitly provided*. ```python import torch from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, AutoConfig from peft import PeftModel, PeftConfig # Specify the path to your saved LoRA model repository # This could be a local path or a Hugging Face Hub repo ID # PEFT_MODEL_PATH = "./bart-lora-finance/best_model" # Local path example PEFT_MODEL_PATH = "deoleojr/bart-finance-lora" # HF Hub example (replace if different) # Determine device device = "cuda" if torch.cuda.is_available() else "cpu" print(f"Using device: {device}") # Load the configuration from the PEFT model path try: config = PeftConfig.from_pretrained(PEFT_MODEL_PATH) print(f"Base model identified from config: {config.base_model_name_or_path}") except Exception as e: print(f"Error loading PeftConfig: {e}") # Fallback or exit if config cannot be loaded config = None # Set config to None or handle error appropriately BASE_MODEL_NAME = "facebook/bart-large-cnn" # Manually specify if needed print(f"Warning: Could not load PeftConfig. Assuming base model: {BASE_MODEL_NAME}") # Load the base model if config: BASE_MODEL_NAME = config.base_model_name_or_path try: print(f"Loading base model: {BASE_MODEL_NAME}...") base_model = AutoModelForSeq2SeqLM.from_pretrained(BASE_MODEL_NAME) print("Base model loaded.") except Exception as e: print(f"Error loading base model '{BASE_MODEL_NAME}': {e}") exit() # Load the PEFT model (LoRA layers) on top of the base model try: print(f"Loading LoRA adapter from: {PEFT_MODEL_PATH}...") # Ensure the base model is loaded before applying adapters model = PeftModel.from_pretrained(base_model, PEFT_MODEL_PATH) model.to(device) # Move the combined model to the device model.eval() # Set model to evaluation mode print("LoRA model loaded and ready.") except Exception as e: print(f"Error loading PEFT model: {e}") exit() # Load the tokenizer (usually saved with the adapters) try: print("Loading tokenizer...") tokenizer = AutoTokenizer.from_pretrained(PEFT_MODEL_PATH) print("Tokenizer loaded.") except Exception as e: print(f"Error loading tokenizer from {PEFT_MODEL_PATH}: {e}") # Fallback to base model tokenizer if necessary try: print(f"Falling back to base model tokenizer: {BASE_MODEL_NAME}") tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_NAME) print("Base model tokenizer loaded.") except Exception as e_base_tok: print(f"Error loading base tokenizer: {e_base_tok}") exit() # Example usage question = "What was the main reason for Tesla's stock rally?" context = "Tesla (TSLA.O) rallied 10% after Morgan Stanley upgraded the electric car maker to 'overweight' from 'equal-weight', saying its Dojo supercomputer could boost the company's market value by nearly $600 billion." # Use the specified prompt format prompt = f"Instruction: {question}\n\n[Context Information]\n{context}" print(f"\nInput Prompt:\n{prompt}") inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=512).to(device) # Generate the answer with torch.no_grad(): # Adjust generation parameters as needed outputs = model.generate( **inputs, max_new_tokens=100, # Max tokens to ADD to the input temperature=0.7, # Controls randomness (lower is more deterministic) top_k=50, # Considers top K tokens top_p=0.95, # Considers tokens cumulative prob > P do_sample=True, # Use sampling (needed for temp, top_k, top_p) num_beams=1 # Use 1 for sampling, >1 for beam search ) prediction = tokenizer.decode(outputs[0], skip_special_tokens=True).strip() print(f"\nQuestion: {question}") print(f"Generated Answer: {prediction}") ``` **Intended Use:** Primarily for offline analysis or integration into systems where context is supplied alongside the question. Suitable for assisting financial analysts or researchers needing fact extraction from specific texts. Not intended for real-time trading decisions or fully automated financial advice due to limitations. ## 6. Prompt Format The model expects prompts formatted with an "Instruction:" prefix for the question, followed by two newlines, the label "[Context Information]", a newline, and then the context text itself. ``` Instruction: {question} [Context Information] {news_article_chunk} ``` Replace {question} with the user's question and {context_passage} with the relevant text snippet. ## 7. Expected Output Format The model outputs a single block of text representing the answer generated based on the provided question and context. ``` A concise, fact-based answer derived from the provided context. Example: "Morgan Stanley upgraded Tesla to 'overweight' from 'equal-weight', citing the potential of its Dojo supercomputer." ``` ## 8. Limitations Context Dependency: Performance heavily relies on the quality and relevance of the provided context. It may struggle or hallucinate if the context is insufficient or irrelevant to the question. Retriever Performance: Benchmarks indicated challenges with retrieving highly relevant context snippets from the financial_phrasebank corpus, which impacts the overall RAG quality regardless of the generator model. The model's performance in a real RAG system will depend heavily on the effectiveness of the retrieval component used. Evaluation Metrics: Automated metrics like BLEU/ROUGE (not used here) and even model-based metrics (Faithfulness, Relevancy) do not perfectly capture factual correctness or nuanced understanding. Human oversight or external fact-checking is advisable for critical applications. Potential Relevancy Decrease: LLM-as-Judge evaluation suggested that fine-tuning, while potentially improving faithfulness, might have slightly reduced the direct relevance of answers compared to the baseline model in some cases.