Spaces:
Running
Running
| title: MT564AITraining | |
| emoji: 🚀 | |
| colorFrom: blue | |
| colorTo: gray | |
| sdk: docker | |
| pinned: false | |
| license: apache-2.0 | |
| short_description: MT564Model training | |
| # SWIFT-MT564-Assistant | |
| Added MT564 TinyLlama training interface | |
| ✓ Created comprehensive training UI with file upload | |
| ✓ Integrated horoscope harvesting with MT564 training | |
| ✓ Both systems running in unified application | |
| ✓ Navigation links connect both functionalities | |
| The application now provides both data harvesting for horoscopes AND MT564 TinyLlama training with a complete UI. You can access the MT564 training interface through the navigation menu. | |
| ## Project Overview | |
| This project creates an AI-powered documentation assistant for financial messaging standards, specifically focused on the SWIFT MT564 message type. It combines web scraping, data processing, TinyLlama fine-tuning, and a user-friendly interface to provide an intelligent assistant for financial messaging professionals. | |
| ## Key Components | |
| ### 1. Data Collection & Processing | |
| - **Web Scraper**: Extracts structured data from [ISO20022 SWIFT MT564 documentation](https://www.iso20022.org/15022/uhb/finmt564.htm) | |
| - **PDF Parser**: Extracts text and structural information from uploaded SWIFT documentation PDFs | |
| - **Data Formatter**: Converts scraped and parsed data into training examples for the model | |
| ### 2. Model Training Pipeline | |
| - **TinyLlama Integration**: Implementation of TinyLlama, a smaller and more efficient LLM | |
| - **Fine-tuning Scripts**: Specialized scripts for training on SWIFT message documentation | |
| - **Evaluation Tools**: Methods to test the model's understanding of SWIFT message formats | |
| ### 3. User Interface | |
| - **Web Application**: Flask-based interface for interacting with the model | |
| - **PDF Upload**: Functionality to upload and process SWIFT documentation PDFs | |
| - **Question-Answering System**: Interactive chat interface for asking questions about MT564 and related formats | |
| ## Technical Architecture | |
| ``` | |
| SWIFT-MT564-Assistant/ | |
| ├── scrapers/ # Web scraping components | |
| │ ├── iso20022_scraper.py # Scraper for ISO20022 website | |
| │ ├── pdf_parser.py # PDF extraction utilities | |
| │ └── data_processor.py # Converts raw data to training format | |
| │ | |
| ├── model/ # ML model components | |
| │ ├── tinyllama_trainer.py # Fine-tuning implementation | |
| │ ├── data_formatter.py # Prepares data for training | |
| │ └── evaluator.py # Tests model performance | |
| │ | |
| ├── webapp/ # Web application | |
| │ ├── app.py # Flask application | |
| │ ├── templates/ # HTML templates | |
| │ │ ├── index.html # Main page | |
| │ │ └── result.html # Results display | |
| │ └── static/ # CSS, JS, and other static files | |
| │ | |
| ├── data/ # Data storage | |
| │ ├── raw/ # Raw scraped data | |
| │ ├── processed/ # Processed training data | |
| │ └── uploaded/ # User-uploaded PDFs | |
| │ | |
| ├── train_mt564_model.py # Script to train the model | |
| ├── requirements.txt # Project dependencies | |
| └── README.md # Project documentation | |
| ``` | |
| ## How It Works | |
| 1. **Data Collection Phase**: | |
| - The ISO20022 scraper extracts structured data from the SWIFT MT564 documentation | |
| - The data is processed and converted into a training dataset of instruction-response pairs | |
| 2. **Model Training Phase**: | |
| - TinyLlama is fine-tuned on the specialized SWIFT message format data | |
| - The model learns the structure, fields, and usage of MT564 messages | |
| 3. **User Interaction Phase**: | |
| - Users upload SWIFT documentation PDFs through the web interface | |
| - The system extracts and processes the PDF content | |
| - Users ask questions about SWIFT messages and receive accurate, contextual responses | |
| ## Installation & Setup | |
| ### Prerequisites | |
| - Python 3.8+ | |
| - PyTorch | |
| - Transformers library | |
| - Flask | |
| - PDF processing libraries | |
| ### Installation Steps | |
| ```bash | |
| # Clone the repository | |
| git clone <repository-url> | |
| cd SWIFT-MT564-Assistant | |
| # Create a virtual environment | |
| python -m venv venv | |
| source venv/bin/activate # On Windows: venv\Scripts\activate | |
| # Install dependencies | |
| pip install -r requirements.txt | |
| # Download and prepare the model | |
| python prepare_mt564_data.py | |
| # Run the web application | |
| python main.py | |
| ``` | |
| ## Usage | |
| ### Training the Model | |
| ```bash | |
| # Run the scraper to collect data | |
| python scrapers/iso20022_scraper.py | |
| # Process the data | |
| python scrapers/data_processor.py | |
| # Train the model | |
| python train_mt564_model.py | |
| ``` | |
| ### Using the Web Interface | |
| 1. Start the Flask application: `python main.py` | |
| 2. Open a browser and navigate to: `http://localhost:5000` | |
| 3. Upload a SWIFT MT564 documentation PDF | |
| 4. Ask questions about the SWIFT message format | |
| ## Future Enhancements | |
| - Expand coverage to additional SWIFT message types (MT565, MT566, etc.) | |
| - Implement multi-document reasoning across different SWIFT standards | |
| - Add support for ISO20022 MX message formats | |
| - Develop specialized modules for message validation and conversion |