{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "89c827e0",
   "metadata": {},
   "source": [
    "# Module 4 Dataset Exploration: Amod Counseling Q&A\n",
    "\n",
    "This notebook explores Hugging Face dataset `Amod/mental_health_counseling_conversations` before using it in retrieval. The goal is to understand the dataset shape, repeated questions, text lengths, and cleaning needs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb5fc3ea",
   "metadata": {},
   "source": [
    "## 1. Setup\n",
    "\n",
    "Keep imports simple and readable. The notebook saves a cleaned JSON file locally and a small summary report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ce99679",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import json\n",
    "import re\n",
    "\n",
    "import pandas as pd\n",
    "pd.set_option(\"display.max_colwidth\",None)\n",
    "from datasets import load_dataset\n",
    "\n",
    "PROJECT_ROOT = Path.cwd()\n",
    "if PROJECT_ROOT.name == \"notebooks\":\n",
    "    PROJECT_ROOT = PROJECT_ROOT.parent\n",
    "PROCESSED_DIR = PROJECT_ROOT / \"data\" / \"processed\"\n",
    "REPORT_DIR = PROJECT_ROOT / \"reports\" / \"module_4_rag_retrieval\"\n",
    "\n",
    "PROCESSED_DIR.mkdir(parents=True, exist_ok=True)\n",
    "REPORT_DIR.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "798eb55b",
   "metadata": {},
   "source": [
    "## 2. Load Dataset\n",
    "\n",
    "The dataset has two columns: `Context` for the user question/problem and `Response` for a counselor-style answer. The cleaned output renames them to `question` and `answer` for readability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e983230",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(\"Amod/mental_health_counseling_conversations\", split=\"train\")\n",
    "df = dataset.to_pandas()\n",
    "\n",
    "print(dataset)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80557ba1",
   "metadata": {},
   "source": [
    "## 3. Rename Columns\n",
    "\n",
    "The original dataset columns are `Context` and `Response`. For easier explanation, we rename them to `question` and `answer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10c79c92",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.rename(columns={\"Context\": \"question\", \"Response\": \"answer\"})\n",
    "print(df.shape)\n",
    "df.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "222ab699",
   "metadata": {},
   "source": [
    "## 4. Basic Cleaning\n",
    "\n",
    "This keeps the original meaning but removes repeated whitespace and strange spacing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a33b8fc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_text(text):\n",
    "    text = str(text)\n",
    "    text = text.replace(\"\\xa0\", \" \")\n",
    "    text = re.sub(r\"([.!?])(?=[A-Z])\", r\"\\1 \", text)\n",
    "    text = re.sub(r\"\\s+\", \" \", text)\n",
    "    return text.strip()\n",
    "\n",
    "df[\"question\"] = df[\"question\"].apply(clean_text)\n",
    "df[\"answer\"] = df[\"answer\"].apply(clean_text)\n",
    "\n",
    "df[[\"question\", \"answer\"]].head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeda7401",
   "metadata": {},
   "source": [
    "## 5. Missing, Empty, And Duplicate Checks\n",
    "\n",
    "This tells us whether the dataset can be used directly or needs filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "380ce114",
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_checks = {\n",
    "    \"raw_rows\": int(len(df)),\n",
    "    \"empty_questions\": int((df[\"question\"] == \"\").sum()),\n",
    "    \"empty_answers\": int((df[\"answer\"] == \"\").sum()),\n",
    "    \"very_short_answers\": int((df[\"answer_words\"] < MIN_ANSWER_WORDS).sum()),\n",
    "    \"exact_duplicate_rows\": int(df.duplicated(subset=[\"question\", \"answer\"]).sum()),\n",
    "    \"unique_questions_before_cleaning\": int(df[\"question\"].nunique()),\n",
    "    \"unique_answers_before_cleaning\": int(df[\"answer\"].nunique()),\n",
    "}\n",
    "\n",
    "quality_checks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "933b7cee",
   "metadata": {},
   "source": [
    "## 6. Text Length Analysis\n",
    "\n",
    "Length matters because long answers may need chunking, while very short answers may not be useful for retrieval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68bce65a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"question_words\"] = df[\"question\"].str.split().str.len()\n",
    "df[\"answer_words\"] = df[\"answer\"].str.split().str.len()\n",
    "\n",
    "length_summary = df[[\"question_words\", \"answer_words\"]].describe().round(2)\n",
    "length_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1516e70",
   "metadata": {},
   "source": [
    "## 7. Repeated Questions\n",
    "\n",
    "This is one of the dataset secrets: many user questions appear more than once with different counselor answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66cc5088",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_counts = df[\"question\"].value_counts()\n",
    "repeated_questions = question_counts[question_counts > 1]\n",
    "\n",
    "print(\"Repeated questions:\", len(repeated_questions))\n",
    "print(\"Rows belonging to repeated questions:\", int(df[\"question\"].isin(repeated_questions.index).sum()))\n",
    "\n",
    "repeated_questions.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c2e6fa",
   "metadata": {},
   "source": [
    "## 8. Example Of A Repeated Question\n",
    "\n",
    "This shows why the dataset is not a typical document corpus. One user problem can have many possible answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffde1205",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_question = repeated_questions.index[0]\n",
    "example_rows = df[df[\"question\"] == example_question][[\"question\", \"answer\"]]\n",
    "\n",
    "print(\"Number of answers for this question:\", len(example_rows))\n",
    "print(\"Question:\\n\", example_question[:1000])\n",
    "\n",
    "example_rows[\"answer\"].head(5).to_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67da0b44",
   "metadata": {},
   "source": [
    "## 9. Short And Long Answers\n",
    "\n",
    "Short answers may be weak retrieval documents. Very long answers may need chunking later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c48cd8ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "short_answers = df.sort_values(\"answer_words\").head(5)[[\"question\", \"answer\", \"answer_words\"]]\n",
    "long_answers = df.sort_values(\"answer_words\", ascending=False).head(5)[[\"question\", \"answer\", \"answer_words\"]]\n",
    "\n",
    "short_answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b073a88",
   "metadata": {},
   "outputs": [],
   "source": [
    "long_answers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be262b06",
   "metadata": {},
   "source": [
    "## 10. Build A Clean Q&A Dataset\n",
    "\n",
    "For now, keep one row per Q&A pair. Add IDs and simple metadata. Later we can decide whether to group repeated contexts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e24fe5d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "MIN_ANSWER_WORDS = 25\n",
    "\n",
    "clean_df = df.copy()\n",
    "clean_df = clean_df[clean_df[\"question\"] != \"\"]\n",
    "clean_df = clean_df[clean_df[\"answer\"] != \"\"]\n",
    "clean_df = clean_df[clean_df[\"answer_words\"] >= MIN_ANSWER_WORDS]\n",
    "clean_df = clean_df.drop_duplicates(subset=[\"question\", \"answer\"]).reset_index(drop=True)\n",
    "\n",
    "question_group_sizes = clean_df[\"question\"].value_counts()\n",
    "clean_df[\"question_group_size\"] = clean_df[\"question\"].map(question_group_sizes)\n",
    "clean_df[\"qa_id\"] = [f\"amod_qa_{i:04d}\" for i in range(1, len(clean_df) + 1)]\n",
    "clean_df[\"source\"] = \"Amod/mental_health_counseling_conversations\"\n",
    "\n",
    "clean_df = clean_df[[\n",
    "    \"qa_id\",\n",
    "    \"source\",\n",
    "    \"question\",\n",
    "    \"answer\",\n",
    "    \"question_words\",\n",
    "    \"answer_words\",\n",
    "    \"question_group_size\",\n",
    "]]\n",
    "\n",
    "clean_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3adb152",
   "metadata": {},
   "source": [
    "## 11. Save Clean Dataset And Summary\n",
    "\n",
    "The processed dataset is saved locally under `data/processed`. The summary report is saved under `reports/module_4_rag_retrieval`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79ae675f",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_path = PROCESSED_DIR / \"amod_clean_qa.json\"\n",
    "summary_path = REPORT_DIR / \"amod_dataset_summary.json\"\n",
    "\n",
    "clean_records = clean_df.to_dict(orient=\"records\")\n",
    "clean_path.write_text(json.dumps(clean_records, indent=2, ensure_ascii=False), encoding=\"utf-8\")\n",
    "\n",
    "summary = {\n",
    "    \"dataset\": \"Amod/mental_health_counseling_conversations\",\n",
    "    \"raw_rows\": int(len(df)),\n",
    "    \"clean_rows\": int(len(clean_df)),\n",
    "    \"quality_checks\": quality_checks,\n",
    "    \"unique_questions\": int(clean_df[\"question\"].nunique()),\n",
    "    \"unique_answers\": int(clean_df[\"answer\"].nunique()),\n",
    "    \"repeated_question_count\": int((question_group_sizes > 1).sum()),\n",
    "    \"rows_with_repeated_question\": int((clean_df[\"question_group_size\"] > 1).sum()),\n",
    "    \"min_question_words\": int(clean_df[\"question_words\"].min()),\n",
    "    \"max_question_words\": int(clean_df[\"question_words\"].max()),\n",
    "    \"average_question_words\": round(float(clean_df[\"question_words\"].mean()), 2),\n",
    "    \"min_answer_words\": int(clean_df[\"answer_words\"].min()),\n",
    "    \"max_answer_words\": int(clean_df[\"answer_words\"].max()),\n",
    "    \"average_answer_words\": round(float(clean_df[\"answer_words\"].mean()), 2),\n",
    "    \"minimum_answer_words_filter\": MIN_ANSWER_WORDS,\n",
    "    \"top_repeated_question_group_sizes\": [int(value) for value in question_group_sizes.head(10).to_list()],\n",
    "    \"main_observation\": \"The dataset is Q&A/counseling-case shaped, with many repeated questions and multiple possible answers per user concern.\",\n",
    "}\n",
    "\n",
    "summary_path.write_text(json.dumps(summary, indent=2, ensure_ascii=False), encoding=\"utf-8\")\n",
    "\n",
    "print(f\"Saved clean dataset to: {clean_path}\")\n",
    "print(f\"Saved summary report to: {summary_path}\")\n",
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc869958",
   "metadata": {},
   "source": [
    "## 12. Practical Interpretation\n",
    "\n",
    "This dataset should not be treated like a normal document corpus. It is better understood as a collection of counseling cases. For retrieval, there are two reasonable options:\n",
    "\n",
    "1. Keep one Q&A pair per record and retrieve similar cases.\n",
    "2. Group repeated questions and summarize answer patterns later.\n",
    "\n",
    "For the first Module 4 version, keep the cleaned Q&A records simple. Later, combine them with the CCI information-sheet corpus for richer retrieval."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}