implement core ML pipeline and unified Streamlit app for data and modeling tasks.

.gitignore

@@ -7,3 +7,11 @@ __pycache__/
 *.pyd
 .Python
 streamlit_results/
+src/enhanced_preprocessing_results/
+ml_experiment_results/
+
+*.dill
+*.tsv
+catboost_info/
+predictions_*
+models_export_*

README.md

@@ -1,377 +1,226 @@
 ---
-title: ML Pipeline PRO Dashboard
-emoji: 🤖
-colorFrom: purple
+title: TimeFlow Pro
+emoji: 📊
+colorFrom: blue
 colorTo: indigo
-sdk: streamlit
+sdk: docker
 pinned: true
 app_file: app.py
-sdk_version: 1.53.1
-license: mit
-short_description: Full ML pipeline dashboard with 28+ algorithms and ensembles
+sdk_version: 1.52.2
 ---
 
-# 🤖 ML Pipeline PRO Dashboard
+# 📊 TimeFlow Pro
 
 <div align="center">
 
-**Complete Machine Learning Pipeline with 28+ Algorithms - From Data to Deployment**
+**Intelligent Time Series Data Analysis and Preprocessing Platform**
 
-*Advanced ML workflow automation with full model comparison and ensemble capabilities*
+*Advanced pipeline for data preparation and feature engineering*
 
-[![Hugging Face](https://img.shields.io/badge/🤗-Hugging%20Face%20Space-blue)](https://huggingface.co/spaces/your-username/ml-pipeline-pro-dashboard)
+[![Hugging Face](https://img.shields.io/badge/🤗-Hugging%20Face%20Space-blue)](https://huggingface.co/spaces/your-username/timeflow-pro)
 [![Streamlit](https://img.shields.io/badge/Interface-Streamlit-FF4B4B)](https://streamlit.io)
-[![Python](https://img.shields.io/badge/Python-3.8+-blue)](https://python.org)
-[![Scikit-learn](https://img.shields.io/badge/ML-Scikit--learn-orange)](https://scikit-learn.org)
+[![Python](https://img.shields.io/badge/Python-3.9+-blue)](https://python.org)
 
 </div>
 
-## 🚀 Overview
+## 🌟 Overview
 
-**ML Pipeline PRO Dashboard** is a comprehensive web application that provides a complete machine learning workflow in one interface. It seamlessly integrates 28+ regression algorithms with advanced preprocessing, hyperparameter tuning, ensemble methods, and model deployment capabilities. Designed for data scientists, ML engineers, and analysts, it eliminates the complexity of switching between multiple tools.
+TimeFlow Pro is a comprehensive platform for time series data analysis, preprocessing, and feature engineering. Designed for data scientists and analysts, it provides an intuitive interface for transforming raw time series data into ML-ready datasets with advanced preprocessing capabilities.
 
-## ✨ Key Features
+## 🚀 Key Features
 
-### 📊 **Complete ML Workflow**
-- **End-to-End Pipeline**: Data loading → preprocessing → model training → evaluation → deployment
-- **28+ Regression Algorithms**: From linear models to gradient boosting ensembles
-- **Interactive Dashboard**: Real-time visualization of model performance
-- **One-Click Operations**: Complex workflows simplified to single clicks
+### 📈 **Data Analysis & Visualization**
+- **Interactive Data Exploration**: Real-time preview and statistics
+- **Missing Value Analysis**: Smart detection and handling strategies
+- **Outlier Detection**: Multiple methods including IQR, Z-Score, Isolation Forest
+- **Temporal Analysis**: Seasonality detection, trend analysis, decomposition
 
-### 🤖 **Advanced Model Library**
-- **Linear Models**: Linear Regression, Ridge, Lasso, ElasticNet, Bayesian Ridge
-- **Tree-Based Models**: Random Forest, XGBoost, LightGBM, CatBoost, Gradient Boosting
-- **Ensemble Methods**: Voting, Stacking, Weighted Averaging, Bagging
-- **Advanced Algorithms**: SVR, MLP, Gaussian Process, Theil-Sen, RANSAC
+### ⚙️ **Advanced Preprocessing Pipeline**
+- **Feature Engineering**: Automatic lag features, rolling statistics, seasonal components
+- **Stationarity Checking**: ADF tests and transformation suggestions
+- **Data Scaling**: Robust, Standard, MinMax, and custom scaling methods
+- **Feature Selection**: Correlation, variance, mutual information, RF importance
 
-### ⚙️ **Smart Automation**
-- **Auto Feature Engineering**: Automatic handling of missing values, scaling, feature selection
-- **Hyperparameter Tuning**: Grid Search, Random Search with intelligent defaults
-- **Model Comparison**: Side-by-side performance metrics and visualization
-- **Best Model Selection**: Automatic identification of top-performing algorithms
-
-### 📈 **Comprehensive Analytics**
-- **Real-time Metrics**: RMSE, MAE, R², MAPE with interactive charts
-- **Feature Importance**: Visual analysis of model decision factors
-- **Error Analysis**: Residual plots, prediction vs actual visualization
-- **Performance Benchmarks**: Model comparison across multiple metrics
+### 🏗️ **ML-Ready Outputs**
+- **Train/Validation/Test Splits**: Time-based or random splitting
+- **Multiple Export Formats**: CSV, Parquet, Excel, JSON
+- **Model Integration**: Ready-to-use datasets for scikit-learn, XGBoost, LightGBM
+- **Visual Reports**: Comprehensive pipeline execution reports
 
 ## 🎮 Quick Start
 
-### 1. **Launch the Dashboard**
-```bash
-# Option 1: Use Hugging Face Space (Recommended)
-https://huggingface.co/spaces/your-username/ml-pipeline-pro-dashboard
+### 1. **Upload Your Data**
+- Support for CSV, Excel, Parquet formats
+- Automatic date parsing and validation
+- Smart column type detection
 
-# Option 2: Run locally
-streamlit run app.py
-```
-
-### 2. **Load Your Data**
-- **Upload CSV/Excel/Parquet files** or use **demo data**
-- Automatic data validation and schema detection
-- Support for time series and tabular data
-
-### 3. **Configure Your Experiment**
+### 2. **Configure Pipeline**
 ```python
-# Auto-configured settings based on your data
+# Example configuration
 config = {
-    'target_column': 'auto-detected',
+    'target_column': 'sales',
     'test_size': 0.2,
-    'models': 'all_algorithms',
-    'tuning': 'auto',
-    'ensembles': 'enabled'
+    'max_lags': 5,
+    'seasonal_period': 365,
+    'scaling_method': 'robust'
 }
 ```
 
-### 4. **Train & Compare Models**
-- One-click training of all 28+ algorithms
-- Real-time progress tracking
-- Interactive comparison dashboard
-- Export results with one click
+### 3. **Run Pipeline & Export**
+- Execute full preprocessing pipeline
+- Download processed data
+- Get feature importance reports
+- Export modeling datasets
 
-## 🏗️ Technical Architecture
+## 📊 Technical Architecture
 
-### 🔧 **Pipeline Structure**
+### 🔧 **Pipeline Components**
 ```
-Data Layer → Preprocessing → Feature Engineering → Model Training
+Data Loading → Validation → Missing Handling → Outlier Treatment
      ↓
-Hyperparameter Tuning → Ensemble Creation → Evaluation → Deployment
-```
-
-### 📦 **Core Components**
-- **Model Factory**: Dynamic creation of 28+ sklearn-compatible models
-- **Validator**: Smart model compatibility checking
-- **Visualizer**: Interactive Plotly-based analytics
-- **Exporter**: Multi-format model serialization
-
-### 🎯 **Model Categories**
-```
-📈 Linear Models (13 algorithms)
-🌳 Tree & Ensemble Models (8 algorithms)
-🚀 Gradient Boosting (3 algorithms)
-🤖 Advanced Methods (4 algorithms)
+Feature Engineering → Stationarity Check → Correlation Analysis
+     ↓
+Data Splitting → Scaling → Feature Selection → Final Validation
 ```
 
-## 📊 Feature Details
-
-### **Data Preparation**
-- Smart missing value imputation (mean, median, mode)
-- Automatic feature scaling (Standard, MinMax, Robust)
-- Outlier detection and treatment
-- Feature selection and dimensionality reduction
-
-### **Model Training**
-- Batch training of all selected models
-- Memory-efficient processing
-- Progress tracking with visual indicators
-- Automatic error handling and fallbacks
-
-### **Evaluation & Comparison**
-- Interactive metrics dashboard
-- Feature importance visualization
-- Residual analysis and error distribution
-- Model performance heatmaps
-
-### **Advanced Features**
-- Hyperparameter optimization (Grid/Random Search)
-- Multiple ensemble methods (Voting, Stacking, Weighted)
-- Model persistence (Pickle, Joblib, Dill)
-- Comprehensive export capabilities
+### 🏆 **Core Features**
+- **Multi-stage Validation**: Raw, processed, and final data validation
+- **Memory Optimization**: Efficient handling of large datasets
+- **Error Recovery**: Graceful handling of pipeline failures
+- **Reproducible Results**: Configuration saving and logging
+
+## 📚 Use Cases
+
+### 🏢 **Business Analytics**
+- Sales forecasting and trend analysis
+- Inventory optimization
+- Customer behavior prediction
+- Financial time series analysis
+
+### 🏭 **Industrial Applications**
+- Sensor data preprocessing
+- Predictive maintenance
+- Quality control monitoring
+- Energy consumption forecasting
+
+### 🎓 **Academic Research**
+- Time series modeling experiments
+- Feature engineering research
+- Algorithm comparison studies
+- Educational tool for data science
 
-## 🎯 Use Cases
+## 🛠️ Installation
 
-### 🔬 **Research & Development**
-- Algorithm comparison studies
-- Feature engineering experiments
-- Hyperparameter optimization research
-- New model benchmarking
-
-### 🏢 **Business Applications**
-- Sales forecasting
-- Customer churn prediction
-- Financial modeling
-- Demand planning
-- Quality prediction
-
-### 🎓 **Education & Training**
-- ML algorithm visualization
-- Hands-on model comparison
-- Feature importance demonstration
-- Ensemble method teaching
-
-### ⚡ **Prototyping & Proof-of-Concept**
-- Rapid model development
-- Quick performance benchmarking
-- Business case validation
-- MVP development
-
-## 🛠️ Installation & Deployment
-
-### **Local Installation**
+### Local Development
 ```bash
 # Clone repository
-git clone https://huggingface.co/spaces/your-username/ml-pipeline-pro-dashboard
-cd ml-pipeline-pro-dashboard
+git clone https://huggingface.co/spaces/your-username/timeflow-pro
+cd timeflow-pro
 
-# Install requirements
+# Install dependencies
 pip install -r requirements.txt
 
-# Launch application
+# Run application
 streamlit run app.py
 ```
 
-### **Docker Deployment**
-```dockerfile
-FROM python:3.9-slim
-
-WORKDIR /app
-COPY . .
-
-RUN pip install --no-cache-dir -r requirements.txt
-
-EXPOSE 8501
-CMD ["streamlit", "run", "app.py", "--server.port=8501"]
-```
-
-### **Cloud Deployment**
-- **Hugging Face Spaces**: One-click deployment
-- **Streamlit Cloud**: Native Streamlit hosting
-- **AWS/Azure/GCP**: Container-based deployment
-- **Heroku**: Simple web app deployment
-
-## 📈 Performance Metrics
-
-### **Training Speed**
-| Model Count | Training Time | Memory Usage |
-|-------------|---------------|--------------|
-| 5 models    | ~30 seconds   | <500 MB      |
-| 15 models   | ~2 minutes    | <1 GB        |
-| 28 models   | ~5 minutes    | <1.5 GB      |
-
-### **Model Accuracy**
-| Dataset Size | Best R² Range | Typical RMSE Improvement |
-|--------------|---------------|-------------------------|
-| Small (<1K)  | 0.6-0.8       | 15-25%                  |
-| Medium (1K-10K) | 0.7-0.9    | 20-30%                  |
-| Large (>10K) | 0.8-0.95      | 25-35%                  |
-
-## 🔧 Configuration Options
-
-### **Data Settings**
-```yaml
-data:
-  target_column: auto_detect
-  test_size: 0.2
-  validation_size: 0.1
-  random_state: 42
-  shuffle: true
-```
-
-### **Model Settings**
-```yaml
-models:
-  baseline_models: all
-  enable_ensembles: true
-  ensemble_methods: [voting, stacking, weighted]
-  top_n_for_ensemble: 5
-```
-
-### **Training Settings**
-```yaml
-training:
-  use_advanced_pipeline: true
-  enable_cross_validation: true
-  cv_folds: 5
-  n_jobs: 1
-  early_stopping: true
-```
+### Docker Deployment
+```bash
+# Build Docker image
+docker build -t timeflow-pro .
 
-### **Tuning Settings**
-```yaml
-tuning:
-  enable_hyperparameter_tuning: true
-  tuning_method: grid
-  n_trials: 30
-  cv_folds: 3
+# Run container
+docker run -p 8501:8501 timeflow-pro
 ```
 
-## 📋 Supported Models
-
-### **Linear Regression Family**
-- Linear Regression
-- Ridge Regression
-- Lasso Regression
-- ElasticNet
-- Bayesian Ridge
-- Huber Regressor
-- SGD Regressor
-- Passive Aggressive Regressor
-- RANSAC Regressor
-- Theil-Sen Regressor
-- Orthogonal Matching Pursuit
-
-### **Tree & Ensemble Models**
-- Random Forest
-- Extra Trees
-- Decision Tree
-- Gradient Boosting
-- Hist Gradient Boosting
-- AdaBoost
-- Bagging Regressor
-
-### **Gradient Boosting**
-- XGBoost
-- LightGBM
-- CatBoost
-
-### **Advanced Methods**
-- Support Vector Regression (SVR)
-- Linear SVR
-- K-Nearest Neighbors
-- Multi-layer Perceptron
-- Gaussian Process
-
-## 🌐 API Integration
+## 🌐 API Usage Example
 
 ```python
-# Example: Using the pipeline programmatically
-from ml_pipeline_dashboard import StreamlitMLPipeline
+from timeflow_pro import TimeFlowPipeline
+import pandas as pd
 
-# Initialize pipeline
-pipeline = StreamlitMLPipeline()
-pipeline.initialize()
+# Load your data
+data = pd.read_csv('your_data.csv')
 
-# Load and prepare data
-pipeline.load_data('demo')
-pipeline.prepare_features()
-
-# Train all models
-results = pipeline.train_models(pipeline.get_all_model_names())
+# Configure pipeline
+config = {
+    'target_column': 'target',
+    'test_size': 0.2,
+    'max_lags': 7,
+    'seasonal_period': 30
+}
 
-# Get best model
-best_model = pipeline.best_model
-predictions = pipeline.predict(new_data, [best_model])
+# Create and run pipeline
+pipeline = TimeFlowPipeline(config)
+processed_data = pipeline.run(data)
 
-# Export results
-pipeline.export_results('export_directory')
+# Get modeling data
+modeling_data = pipeline.get_modeling_data()
+X_train, y_train = modeling_data['X_train'], modeling_data['y_train']
 ```
 
-## 📊 Visualization Examples
-
-### **Model Comparison Dashboard**
-- Interactive bar charts for RMSE, MAE, R²
-- Training time comparison
-- Feature importance heatmaps
-- Prediction error distributions
+## 📈 Performance Benchmarks
 
-### **Real-time Analytics**
-- Live training progress
-- Instant metric updates
-- Dynamic chart rendering
-- Export-ready visualizations
+| Dataset Size | Processing Time | Memory Usage | Features Generated |
+|--------------|----------------|--------------|-------------------|
+| 10K rows | ~5 seconds | <500 MB | 50-100 features |
+| 100K rows | ~30 seconds | <1 GB | 100-200 features |
+| 1M rows | ~5 minutes | <2 GB | 200-500 features |
 
-## 🔒 Security & Privacy
+## 🔧 Configuration Options
 
-### **Data Protection**
-- Local processing only (no data sent externally)
-- Session-based data isolation
-- Automatic temporary file cleanup
-- No persistent storage of uploaded data
+### **Data Processing**
+- `missing_threshold`: Threshold for column removal (0.0-0.5)
+- `outlier_method`: IQR, Z-Score, or Isolation Forest
+- `scaling_method`: Robust, Standard, MinMax, or None
+
+### **Feature Engineering**
+- `max_lags`: Maximum lag features (1-20)
+- `seasonal_period`: Seasonal window (7, 30, 90, 365)
+- `rolling_windows`: List of rolling windows [7, 30, 90]
+
+### **Model Preparation**
+- `feature_selection_method`: Correlation, Variance, RF, Mutual Info
+- `max_features`: Maximum features to select (5-100)
+- `split_method`: Time-based or random splitting
+
+## 📋 Requirements
+
+### **Core Dependencies**
+```txt
+streamlit>=1.28.0
+pandas>=2.0.0
+numpy>=1.24.0
+plotly>=5.17.0
+scikit-learn>=1.3.0
+```
 
-### **Best Practices**
-- Memory-efficient processing
-- Secure model serialization
-- Input validation and sanitization
-- Error handling without data leakage
+### **Optional Dependencies**
+```txt
+xgboost>=2.0.0      # For XGBoost feature importance
+lightgbm>=4.0.0     # For LightGBM integration
+statsmodels>=0.14.0 # For advanced time series analysis
+```
 
 ## 🤝 Contributing
 
-We welcome contributions from the community!
-
-### **How to Contribute**
-1. **Fork** the repository
-2. **Create** a feature branch
-3. **Implement** your changes
-4. **Test** thoroughly
-5. **Submit** a pull request
+We welcome contributions! Here's how you can help:
 
 ### **Areas for Contribution**
-- New model algorithms
-- Additional visualization types
-- Performance optimizations
-- Documentation improvements
-- Bug fixes and enhancements
+1. **New Feature Engineering Methods**
+2. **Additional Visualization Types**
+3. **Export Format Support**
+4. **Performance Optimizations**
+5. **Documentation Improvements**
 
-### **Development Setup**
+### **Development Workflow**
 ```bash
-# Set up development environment
-git clone https://huggingface.co/spaces/your-username/ml-pipeline-pro-dashboard
-cd ml-pipeline-pro-dashboard
-python -m venv venv
-source venv/bin/activate  # On Windows: venv\Scripts\activate
-pip install -r requirements.txt
-pip install -r requirements-dev.txt  # Development dependencies
+# 1. Fork the repository
+# 2. Create feature branch
+git checkout -b feature/new-feature
+
+# 3. Make changes and test
+# 4. Submit pull request
 ```
 
 ## 📜 License
@@ -380,67 +229,40 @@ This project is licensed under the **MIT License** - see the [LICENSE](LICENSE)
 
 ## 🙏 Acknowledgments
 
-### **Core Technologies**
-- **Streamlit** for the amazing web framework
-- **Scikit-learn** for comprehensive ML algorithms
-- **Plotly** for interactive visualizations
-- **Hugging Face** for hosting and community
+### **Special Thanks To:**
+- **Streamlit Team** for the amazing framework
+- **Hugging Face** for hosting the Space
+- **Open Source Community** for invaluable libraries
+- **All Contributors** who helped improve TimeFlow Pro
 
-### **Special Thanks**
-- All open-source contributors
-- The data science community
-- Early testers and feedback providers
-- Everyone who helped improve this tool
+### **Built With:**
+- 🐍 Python
+- 📊 Streamlit
+- 🎨 Plotly
+- 🔧 Scikit-learn
+- 📈 Pandas & NumPy
 
 ## 📞 Support & Contact
 
-### **Get Help**
+### **Get Help:**
 - 📧 **Email**: cool.araby@gmail.com
-- 💬 **Issues**: [Report on Hugging Face](https://huggingface.co/spaces/your-username/ml-pipeline-pro-dashboard/discussions)
-- 🐦 **Twitter**: [@YourHandle](https://twitter.com/yourhandle)
-
-### **Community**
-- ⭐ **Star** the Space if you find it useful
-- 🔄 **Share** with colleagues
-- 💡 **Suggest** new features
-- 🐛 **Report** issues
+- 💬 **Issues**: [GitHub Issues](https://github.com/your-username/timeflow-pro/issues)
+- 💡 **Discussions**: [Community Forum](https://github.com/your-username/timeflow-pro/discussions)
 
-### **Stay Updated**
-- Watch for updates on Hugging Face
-- Follow for new feature announcements
-- Join the community discussions
+### **Stay Updated:**
+- ⭐ **Star** the repository
+- 👁️ **Watch** for releases
+- 🔔 **Enable notifications**
 
 ---
 
 <div align="center">
 
-**Transform Your ML Workflow with One Dashboard**
+**Transform Your Time Series Data with Ease**
 
-*From Data to Deployment - 28+ Algorithms, Infinite Possibilities*
+*TimeFlow Pro - Making Data Preparation Simple and Powerful*
 
 [![Follow on Hugging Face](https://img.shields.io/badge/Follow%20on-🤗%20Hugging%20Face-yellow)](https://huggingface.co/your-username)
-[![Try on Hugging Face](https://img.shields.io/badge/Try%20It-🤗%20Space-blue)](https://huggingface.co/spaces/your-username/ml-pipeline-pro-dashboard)
-
-**Empower Your Data Science Projects with Complete ML Automation**
-
-</div>
+[![GitHub Stars](https://img.shields.io/github/stars/your-username/timeflow-pro?style=social)](https://github.com/your-username/timeflow-pro)
 
-## 🎯 Pro Tips
-
-### **For Best Results**
-1. **Start with demo data** to understand the workflow
-2. **Use auto-configuration** for initial experiments
-3. **Compare all models** before selecting the best
-4. **Experiment with ensembles** for improved accuracy
-5. **Export and save** your best configurations
-
-### **Performance Optimization**
-- Use appropriate data size for your needs
-- Enable parallel processing for large datasets
-- Utilize GPU acceleration when available
-- Monitor memory usage during training
-
-### **Workflow Recommendations**
-1. **Exploratory Phase**: Use demo data, test all models
-2. **Optimization Phase**: Tune hyperparameters, create ensembles
-3. **Production Phase**: Export best model, deploy with confidence
+</div>

fix_streamlit_api.py

@@ -0,0 +1,48 @@
+# fix_streamlit_api.py
+import os
+from pathlib import Path
+
+def replace_in_file(filepath):
+    """Заменяет use_container_width в файле"""
+    with open(filepath, 'r', encoding='utf-8') as f:
+        content = f.read()
+    
+    # Заменяем use_container_width=True
+    new_content = content.replace('use_container_width=True', "width='stretch'")
+    
+    # Заменяем use_container_width=False (если есть)
+    new_content = new_content.replace('use_container_width=False', "width='content'")
+    
+    if content != new_content:
+        with open(filepath, 'w', encoding='utf-8') as f:
+            f.write(new_content)
+        return True
+    return False
+
+def main():
+    """Исправляет все файлы в проекте"""
+    current_dir = Path(__file__).parent
+    
+    # Файлы для исправления
+    files_to_fix = [
+        current_dir / "app.py",
+        current_dir / "streamlit" / "preprocessor_app.py",
+        current_dir / "streamlit" / "ml_app.py"
+    ]
+    
+    fixed_count = 0
+    for filepath in files_to_fix:
+        if filepath.exists():
+            print(f"Fixing {filepath.name}...")
+            if replace_in_file(filepath):
+                fixed_count += 1
+                print(f"  ✓ Fixed")
+            else:
+                print(f"  No changes needed")
+        else:
+            print(f"  File not found: {filepath}")
+    
+    print(f"\n✅ Fixed {fixed_count} files")
+
+if __name__ == "__main__":
+    main()

pipeline/main_pipeline.py → main_pipeline.py

@@ -7,23 +7,22 @@ import os
 import traceback
 from typing import Any, Dict, Optional
 from venv import logger
-from config.config import Config
-from correlations.correlation_analyzer import CorrelationAnalyzer
-from data_loader.data_loader import DataLoader
-from decomposition.decomposer import TimeSeriesDecomposer
-from feature_selection.feature_selector import FeatureSelector
-from features.feature_engineer import FeatureEngineer
 
-from missing_values.missing_analyzer import MissingValueAnalyser
-from outliers.outlier_analyzer import OutlierAnalyser
-from scaling.data_scaler import DataScaler
-from splitting.data_splitter import DataSplitter
-from stationarity.stationarity_checker import StationarityChecker
-from validation.data_validator import DataValidator
 import pandas as pd
 import numpy as np
-
-from visualization.visualization_manager import VisualisationManager
+from src.enhanced_preprocessing.config.config import Config
+from src.enhanced_preprocessing.correlations.correlation_analyzer import CorrelationAnalyzer
+from src.enhanced_preprocessing.data_loader.data_loader import DataLoader
+from src.enhanced_preprocessing.decomposition.decomposer import TimeSeriesDecomposer
+from src.enhanced_preprocessing.feature_selection.feature_selector import FeatureSelector
+from src.enhanced_preprocessing.features.feature_engineer import FeatureEngineer
+from src.enhanced_preprocessing.missing_values.missing_analyzer import MissingValueAnalyser
+from src.enhanced_preprocessing.outliers.outlier_analyzer import OutlierAnalyser
+from src.enhanced_preprocessing.scaling.data_scaler import DataScaler
+from src.enhanced_preprocessing.splitting.data_splitter import DataSplitter
+from src.enhanced_preprocessing.stationarity.stationarity_checker import StationarityChecker
+from src.enhanced_preprocessing.validation.data_validator import DataValidator
+from src.enhanced_preprocessing.visualization.visualization_manager import VisualisationManager
 
 class EnhancedDataPreprocessingPipeline:
     """Enhanced main data preprocessing pipeline"""

{correlations → src/enhanced_preprocessing/correlations}/correlation_analyzer.py

@@ -6,10 +6,12 @@ import traceback
 from typing import Any, Dict, List, Optional
 from venv import logger
 
-from config.config import Config
+
 import numpy as np
 import pandas as pd
 
+from src.enhanced_preprocessing.config.config import Config
+
 class CorrelationAnalyzer:
     """Class for comprehensive correlation and multicollinearity analysis"""
     

{data_loader → src/enhanced_preprocessing/data_loader}/data_loader.py

@@ -7,10 +7,12 @@ import json
 import traceback
 from typing import Dict, List, Optional
 from venv import logger
-from config.config import Config, DataType
+
 import numpy as np
 import pandas as pd
 
+from src.enhanced_preprocessing.config.config import Config, DataType
+
 class DataLoader:
     """Class for loading and initial data processing"""
     

{decomposition → src/enhanced_preprocessing/decomposition}/decomposer.py

@@ -5,11 +5,11 @@ import traceback
 from typing import Dict, Optional
 from venv import logger
 
-from config.config import Config
 
 import pandas as pd
 import numpy as np
 import matplotlib.pyplot as plt
+from src.enhanced_preprocessing.config.config import Config
 import statsmodels.api as sm
 from scipy import stats
 from statsmodels.tsa.seasonal import seasonal_decompose, STL

{feature_selection → src/enhanced_preprocessing/feature_selection}/feature_selector.py

@@ -3,8 +3,8 @@
 # ============================================
 from typing import Dict, List, Optional, Tuple
 from venv import logger
-from config.config import Config
 
+from src.enhanced_preprocessing.config.config import Config
 try:
     import pandas as pd
     import numpy as np

{features → src/enhanced_preprocessing/features}/feature_engineer.py

@@ -3,12 +3,9 @@
 # ============================================
 from typing import Dict, List, Optional
 from venv import logger
-
-from config.config import Config
-
 import pandas as pd
 import numpy as np
-
+from src.enhanced_preprocessing.config.config import Config
 
 class FeatureEngineer:
     """Class for creating new features for time series"""

{missing_values → src/enhanced_preprocessing/missing_values}/missing_analyzer.py

@@ -4,8 +4,8 @@
 from typing import Dict, Tuple
 from venv import logger
 
-from config.config import Config
 from scipy.interpolate import interp1d
+from src.enhanced_preprocessing.config.config import Config
 from statsmodels.tsa.seasonal import seasonal_decompose, STL
 try:
     import pandas as pd

{outliers → src/enhanced_preprocessing/outliers}/outlier_analyzer.py

@@ -4,13 +4,13 @@
 from typing import Dict, List, Tuple
 from venv import logger
 
-from config.config import Config
 import pandas as pd
 import numpy as np
 import matplotlib.pyplot as plt
 from sklearn.neighbors import LocalOutlierFactor
 from sklearn.covariance import EllipticEnvelope
 from scipy import stats
+from src.enhanced_preprocessing.config.config import Config
 
 class OutlierAnalyser:
     """Class for analysing and handling outliers"""

run_pipeline.py → src/enhanced_preprocessing/run_pipeline.py

@@ -1,9 +1,12 @@
 # ============================================
 # RUN 
 # ============================================
-from config.config import Config
-from pipeline.main_pipeline import EnhancedDataPreprocessingPipeline
+
 import pandas as pd
+import os
+
+from main_pipeline import EnhancedDataPreprocessingPipeline
+from src.enhanced_preprocessing.config.config import Config
 
 if __name__ == "__main__":
     """
@@ -13,7 +16,7 @@ if __name__ == "__main__":
     # Configuration with reasonable parameters
     config = Config(
         data_path='temp_data.csv',
-        results_dir='enhanced_preprocessing_results',
+        results_dir='src/enhanced_preprocessing_results',
         target_column='raskhodvoda',
         start_year=1970,
         end_year=1990,
@@ -56,7 +59,14 @@ if __name__ == "__main__":
             print(f"  X_test: {modeling_data['X_test'].shape}")
             print(f"  Features: {len(modeling_data['feature_names'])}")
         
-        # Save final data
-        processed_data.to_csv('enhanced_preprocessing_results\processed_data\enhanced_final_processed_data.csv', 
-                            index=True if isinstance(processed_data.index, pd.DatetimeIndex) else False)
-        print(f"\n✓ Final data saved to 'enhanced_final_processed_data.csv'")
+        # Save final data using config paths
+        # Create subdirectory if it doesn't exist
+        output_dir = os.path.join(config.results_dir, 'processed_data')
+        os.makedirs(output_dir, exist_ok=True)
+        
+        output_path = os.path.join(output_dir, 'enhanced_final_processed_data.csv')
+        processed_data.to_csv(
+            output_path,
+            index=True if isinstance(processed_data.index, pd.DatetimeIndex) else False
+        )
+        print(f"\n✓ Final data saved to '{output_path}'")

{scaling → src/enhanced_preprocessing/scaling}/data_scaler.py

@@ -4,10 +4,10 @@
 from typing import Dict, List, Optional, Tuple
 from venv import logger
 import pandas as pd
-from config.config import Config
 import numpy as np
 import matplotlib.pyplot as plt
 
+from src.enhanced_preprocessing.config.config import Config
 
 class DataScaler:
     """Class for data scaling and normalisation"""

{splitting → src/enhanced_preprocessing/splitting}/data_splitter.py

@@ -5,10 +5,10 @@ from datetime import datetime
 from typing import Dict, Optional, Tuple
 from venv import logger
 import pandas as pd
-from config.config import Config
 import numpy as np
 import matplotlib.pyplot as plt
 
+from src.enhanced_preprocessing.config.config import Config
 
 class DataSplitter:
     """Class for splitting data into train, validation and test sets"""

{stationarity → src/enhanced_preprocessing/stationarity}/stationarity_checker.py

@@ -3,10 +3,10 @@
 # ============================================
 from typing import Dict, Optional
 from venv import logger
-from config.config import Config
 import pandas as pd
 import numpy as np
 import matplotlib.pyplot as plt
+from src.enhanced_preprocessing.config.config import Config
 from statsmodels.tsa.stattools import adfuller, kpss, acf, pacf
 from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
 

{validation → src/enhanced_preprocessing/validation}/data_validator.py

@@ -6,10 +6,9 @@ import json
 from pathlib import Path
 from typing import Dict, List
 from venv import logger
-
-from config.config import Config
 import pandas as pd
 import numpy as np
+from src.enhanced_preprocessing.config.config import Config
 
 class DataValidator:
     """Class for data quality validation"""

{visualization → src/enhanced_preprocessing/visualization}/visualization_manager.py

@@ -5,17 +5,15 @@ import os
 from datetime import datetime
 import json
 from typing import Dict, List, Optional, Tuple, Union, Any
-
 import pandas as pd
 import numpy as np
+from src.enhanced_preprocessing.config.config import Config
 from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
 import matplotlib.pyplot as plt
 import seaborn as sns
 from scipy.stats import gaussian_kde
 import matplotlib
 matplotlib.use('Agg')  # Use non-display backend
-
-from config.config import Config
 import logging
 
 # Logging setup

src/ml_pipeline/Readme.md

@@ -0,0 +1 @@
+(.venv) PS D:\Science\Narzullo\LastVersionTimeFlowPro\src> python -m ml_pipeline.run_mlpipeline

src/ml_pipeline/combine_files.py

@@ -0,0 +1,127 @@
+import os
+from pathlib import Path
+
+def combine_python_files(source_dir, output_file):
+    """
+    Combine all Python files from specified directory into a single file.
+    
+    Args:
+        source_dir (str): Path to source directory
+        output_file (str): Path to output file
+    """
+    source_path = Path(source_dir)
+    
+    # Check if source directory exists
+    if not source_path.exists():
+        print(f"Error: Directory {source_dir} does not exist")
+        return
+    
+    # Get all Python files recursively
+    python_files = list(source_path.rglob("*.py"))
+    
+    if not python_files:
+        print(f"No Python files found in {source_dir}")
+        return
+    
+    print(f"Found {len(python_files)} Python files")
+    
+    with open(output_file, 'w', encoding='utf-8') as out_f:
+        # Write header
+        out_f.write("#" * 80 + "\n")
+        out_f.write("# COMBINED FILE WITH ALL PYTHON SCRIPTS\n")
+        out_f.write(f"# Directory: {source_dir}\n")
+        out_f.write("#" * 80 + "\n\n")
+        
+        # Process each Python file
+        for py_file in sorted(python_files):
+            relative_path = py_file.relative_to(source_path)
+            
+            out_f.write("\n" + "=" * 80 + "\n")
+            out_f.write(f"# FILE: {relative_path}\n")
+            out_f.write("#" * 80 + "\n\n")
+            
+            try:
+                with open(py_file, 'r', encoding='utf-8') as in_f:
+                    content = in_f.read()
+                    out_f.write(content)
+                
+                if not content.endswith('\n'):
+                    out_f.write('\n')
+                    
+            except Exception as e:
+                out_f.write(f"# Error reading file: {e}\n")
+        
+        # Add project structure at the end
+        out_f.write("\n" + "=" * 80 + "\n")
+        out_f.write("# PROJECT STRUCTURE\n")
+        out_f.write("#" * 80 + "\n\n")
+        
+        # Generate tree structure
+        generate_project_structure(source_path, out_f)
+
+def generate_project_structure(root_dir, output_stream):
+    """
+    Generate project tree structure.
+    
+    Args:
+        root_dir (Path): Root directory
+        output_stream: Output stream for writing
+    """
+    def add_directory(dir_path, prefix=""):
+        """Recursively add directory contents."""
+        try:
+            # Get all items, sorted (directories first, then files)
+            items = sorted(dir_path.iterdir(), 
+                          key=lambda x: (not x.is_dir(), x.name.lower()))
+            
+            for i, item in enumerate(items):
+                is_last = (i == len(items) - 1)
+                connector = "└── " if is_last else "├── "
+                
+                if item.is_dir():
+                    output_stream.write(f"{prefix}{connector}{item.name}/\n")
+                    extension = "    " if is_last else "│   "
+                    add_directory(item, prefix + extension)
+                else:
+                    # Count lines for .py files
+                    if item.suffix == '.py':
+                        try:
+                            with open(item, 'r', encoding='utf-8') as f:
+                                lines = len(f.readlines())
+                            line_info = f" ({lines} lines)"
+                        except:
+                            line_info = " (read error)"
+                    else:
+                        line_info = ""
+                    
+                    output_stream.write(f"{prefix}{connector}{item.name}{line_info}\n")
+                    
+        except PermissionError:
+            output_stream.write(f"{prefix}└── [Access denied]\n")
+
+    output_stream.write(f"{root_dir.name}/\n")
+    add_directory(root_dir)
+
+def main():
+    # Specify paths
+    source_directory = "src/ml_pipeline"
+    output_filename = "combined_ml_pipeline.py"
+    
+    print("Starting file combination...")
+    print(f"Source directory: {source_directory}")
+    print(f"Output file: {output_filename}")
+    
+    # Perform combination
+    combine_python_files(source_directory, output_filename)
+    
+    print(f"\nFiles successfully combined into {output_filename}")
+    print(f"Full path: {os.path.abspath(output_filename)}")
+    
+    # Show statistics
+    if os.path.exists(output_filename):
+        with open(output_filename, 'r', encoding='utf-8') as f:
+            lines = len(f.readlines())
+        print(f"Final file contains: {lines} lines")
+
+if __name__ == "__main__":
+    main()

src/ml_pipeline/config/ml_config.py

@@ -0,0 +1,503 @@
+from datetime import datetime
+import json
+import os
+from pathlib import Path
+from dataclasses import dataclass, field, asdict
+from typing import Dict, List, Tuple, Optional, Any, Union, Callable
+import logging
+
+from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, QuantileTransformer, PowerTransformer
+from sklearn.impute import SimpleImputer, KNNImputer
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class MLConfig:
+    """
+    Configuration for an ML experiment.
+    """
+
+    # Paths
+    preprocessing_results_dir: str = r'src\enhanced_ml_results'
+    ml_results_dir: str = r'src\ml_experiment_results'
+    models_dir: str = r'src\saved_models'
+    feature_store_dir: str = r'src\feature_store'
+
+    # Target variable
+    target_column: str = 'raskhodvoda'
+
+    # Data splitting
+    test_size: float = 0.2
+    validation_size: float = 0.1
+    shuffle_split: bool = True
+    stratify: bool = False
+    time_series_split: bool = True
+
+    baseline_models: List[str] = field(default_factory=lambda: [
+        'linear_regression',
+        'ridge',
+        'lasso',
+        'elasticnet',
+        'random_forest',
+        'xgboost',
+        'lightgbm',
+        'catboost',
+        'gradient_boosting',
+        'hist_gradient_boosting',
+        'extra_trees',
+        'ada_boost',
+        'svr',
+        'linear_svr',
+        'knn',
+        'decision_tree',
+        'mlp',
+        'bayesian_ridge',
+        'huber',
+        'kernel_ridge',
+        'pls',
+        'sgd',
+        'passive_aggressive',
+        'ransac',
+        'theil_sen',
+        'orthogonal_matching_pursuit',
+        'gaussian_process'
+    ])
+
+    # Cross-validation parameters
+    cv_folds: int = 5
+    cv_method: str = 'timeseries'
+    random_state: int = 42
+    groups: Optional[str] = None
+
+    # Evaluation metrics
+    scoring_metrics: List[str] = field(default_factory=lambda: [
+        'neg_mean_squared_error',
+        'neg_mean_absolute_error',
+        'r2',
+        'neg_root_mean_squared_error',
+        'explained_variance',
+        'neg_mean_squared_log_error',
+        'neg_median_absolute_error'
+    ])
+
+    # Hyperparameter tuning
+    enable_hyperparameter_tuning: bool = True
+    tuning_method: str = 'bayesian'
+    n_trials: int = 50
+    timeout_per_trial: int = 300
+    param_search_space: Dict = field(default_factory=dict)
+
+    # Ensembles
+    enable_ensembles: bool = True
+    ensemble_methods: List[str] = field(default_factory=lambda: [
+        'voting',
+        'stacking',
+        'blending',
+        'weighted',
+        'bagging',
+        'boosting'])
+    top_n_for_ensemble: int = 5
+
+    # Feature Engineering
+    feature_engineering: bool = True
+    create_polynomial_features: bool = False
+    polynomial_degree: int = 2
+    create_interaction_features: bool = True
+    use_feature_selection: bool = True
+    feature_selection_method: str = 'importance'
+    n_features_to_select: Union[int, float] = 0.8
+    max_features_limit: int = 100
+
+    # Preprocessing
+    scale_features: bool = True
+    scaler_type: str = 'standard'
+    handle_imbalance: bool = False
+    treat_outliers: bool = True
+    outlier_method: str = 'iqr'
+    impute_missing: bool = True
+    impute_method: str = 'median'
+
+    # Additional options
+    use_pipeline: bool = True
+    use_gpu: bool = False
+    n_jobs: int = -1
+    memory_limit: str = '4G'
+    early_stopping_rounds: int = 50
+    use_cache: bool = True
+    cache_dir: str = '.cache'
+    verbose: bool = True
+    debug: bool = False
+
+    # Visualisation
+    save_plots: bool = True
+    plot_style: str = 'seaborn'
+    dpi: int = 300
+    plot_language: str = 'ru'
+
+    # Saving results
+    save_models: bool = True
+    save_predictions: bool = True
+    save_reports: bool = True
+    save_config: bool = True
+    save_feature_importance: bool = True
+    save_residuals: bool = True
+    format: str = 'all'
+
+    # MLOps
+    use_mlflow: bool = False
+    mlflow_tracking_uri: str = 'mlruns'
+    experiment_name: str = 'ml_experiment'
+    log_artifacts: bool = True
+
+    # Monitoring
+    monitor_resources: bool = True
+    profile_memory: bool = True
+    profile_time: bool = True
+
+    def __post_init__(self):
+        self.create_directories()
+        self.setup_param_search_space()
+        self.setup_scaler()
+        self.setup_imputer()
+        self.validate_parameters()
+
+    def create_directories(self):
+        """
+        Create all required directories.
+        """
+        dirs = [
+            self.ml_results_dir,
+            f'{self.ml_results_dir}/models',
+            f'{self.ml_results_dir}/predictions',
+            f'{self.ml_results_dir}/plots',
+            f'{self.ml_results_dir}/plots/residuals',
+            f'{self.ml_results_dir}/plots/predictions',
+            f'{self.ml_results_dir}/plots/feature_importance',
+            f'{self.ml_results_dir}/plots/learning_curves',
+            f'{self.ml_results_dir}/plots/hyperparameters',
+            f'{self.ml_results_dir}/plots/correlations',
+            f'{self.ml_results_dir}/plots/diagnostics',
+            f'{self.ml_results_dir}/plots/shap',
+            f'{self.ml_results_dir}/reports',
+            f'{self.ml_results_dir}/tables',
+            f'{self.ml_results_dir}/logs',
+            f'{self.ml_results_dir}/cache',
+            f'{self.ml_results_dir}/feature_store',
+            self.models_dir,
+            self.cache_dir,
+            self.feature_store_dir
+        ]
+
+        for directory in dirs:
+            Path(directory).mkdir(parents=True, exist_ok=True)
+
+        logger.info(f"Directories created in {self.ml_results_dir}")
+
+    def validate_parameters(self):
+        """
+        Validate parameter correctness.
+        """
+        if self.test_size + self.validation_size >= 1:
+            raise ValueError("The sum of test_size and validation_size must be less than 1")
+
+        if self.n_jobs == -1:
+            self.n_jobs = os.cpu_count()
+        else:
+            self.n_jobs = min(self.n_jobs, os.cpu_count())
+
+        if self.cv_method not in ['kfold', 'timeseries', 'stratified', 'group', 'loo', 'shuffle']:
+            raise ValueError(f"Unknown CV method: {self.cv_method}")
+
+    def setup_param_search_space(self):
+        """
+        Configure the hyperparameter search space.
+        """
+        self.param_search_space = {
+            'random_forest': {
+                'n_estimators': [100, 200, 300, 500],
+                'max_depth': [10, 20, 30, None],
+                'min_samples_split': [2, 5, 10],
+                'min_samples_leaf': [1, 2, 4],
+                'max_features': ['sqrt', 'log2', None],
+                'bootstrap': [True, False],
+                'max_samples': [0.5, 0.7, 0.9, None]
+            },
+            'xgboost': {
+                'n_estimators': [100, 200, 300, 500],
+                'max_depth': [3, 5, 7, 9],
+                'learning_rate': [0.001, 0.01, 0.05, 0.1, 0.2],
+                'subsample': [0.6, 0.8, 1.0],
+                'colsample_bytree': [0.6, 0.8, 1.0],
+                'gamma': [0, 0.1, 0.2, 0.5],
+                'reg_alpha': [0, 0.1, 0.5, 1.0],
+                'reg_lambda': [0.5, 1, 1.5, 2],
+                'min_child_weight': [1, 3, 5],
+                'grow_policy': ['depthwise', 'lossguide']
+            },
+            'lightgbm': {
+                'n_estimators': [100, 200, 300, 500],
+                'num_leaves': [31, 50, 100, 150],
+                'learning_rate': [0.001, 0.01, 0.05, 0.1],
+                'max_depth': [-1, 5, 10, 15],
+                'min_child_samples': [20, 30, 50, 100],
+                'subsample': [0.8, 0.9, 1.0],
+                'colsample_bytree': [0.8, 0.9, 1.0],
+                'reg_alpha': [0, 0.1, 0.5, 1.0],
+                'reg_lambda': [0, 0.1, 0.5, 1.0],
+                'min_split_gain': [0, 0.1, 0.2],
+                'boosting_type': ['gbdt', 'dart', 'goss']
+            },
+            'catboost': {
+                'iterations': [100, 200, 300, 500],
+                'depth': [4, 6, 8, 10],
+                'learning_rate': [0.001, 0.01, 0.05, 0.1],
+                'l2_leaf_reg': [1, 3, 5, 10],
+                'border_count': [32, 64, 128],
+                'random_strength': [0, 0.1, 0.5, 1.0],
+                'bagging_temperature': [0, 0.5, 1.0],
+                'leaf_estimation_iterations': [1, 5, 10],
+                'grow_policy': ['SymmetricTree', 'Depthwise', 'Lossguide']
+            },
+            'gradient_boosting': {
+                'n_estimators': [100, 200, 300],
+                'learning_rate': [0.01, 0.05, 0.1, 0.2],
+                'max_depth': [3, 5, 7, 9],
+                'min_samples_split': [2, 5, 10],
+                'min_samples_leaf': [1, 2, 4],
+                'subsample': [0.8, 0.9, 1.0],
+                'max_features': ['sqrt', 'log2', None],
+                'loss': ['squared_error', 'absolute_error', 'huber', 'quantile']
+            },
+            'hist_gradient_boosting': {
+                'max_iter': [100, 200, 300],
+                'learning_rate': [0.01, 0.05, 0.1],
+                'max_depth': [None, 5, 10, 15],
+                'min_samples_leaf': [20, 30, 50],
+                'l2_regularization': [0, 0.1, 0.5, 1.0],
+                'max_bins': [128, 200, 255],
+                'loss': ['squared_error', 'absolute_error'],
+                'early_stopping': [True, False]
+            },
+            'svr': {
+                'C': [0.1, 1, 10, 100, 1000],
+                'epsilon': [0.01, 0.1, 0.2, 0.5],
+                'kernel': ['linear', 'rbf', 'poly', 'sigmoid'],
+                'gamma': ['scale', 'auto'] + [0.001, 0.01, 0.1, 1],
+                'degree': [2, 3, 4],
+                'coef0': [0.0, 0.1, 0.5, 1.0]
+            },
+            'nu_svr': {
+                'nu': [0.1, 0.3, 0.5, 0.7, 0.9],
+                'C': [0.1, 1, 10, 100],
+                'kernel': ['linear', 'rbf'],
+                'gamma': ['scale', 'auto'] + [0.001, 0.01, 0.1]
+            },
+            'ridge': {
+                'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0],
+                'solver': ['auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']
+            },
+            'lasso': {
+                'alpha': [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0],
+                'selection': ['cyclic', 'random'],
+                'max_iter': [1000, 5000, 10000]
+            },
+            'elasticnet': {
+                'alpha': [0.0001, 0.001, 0.01, 0.1, 1.0],
+                'l1_ratio': [0.1, 0.3, 0.5, 0.7, 0.9, 1.0],
+                'max_iter': [1000, 5000, 10000]
+            },
+            'bayesian_ridge': {
+                'max_iter': [100, 300, 500, 1000],
+                'alpha_1': [1e-6, 1e-5, 1e-4],
+                'alpha_2': [1e-6, 1e-5, 1e-4],
+                'lambda_1': [1e-6, 1e-5, 1e-4],
+                'lambda_2': [1e-6, 1e-5, 1e-4],
+                'compute_score': [True, False]
+            },
+            'huber': {
+                'epsilon': [1.1, 1.35, 1.5, 2.0],
+                'alpha': [0.0001, 0.001, 0.01, 0.1],
+                'max_iter': [100, 500, 1000]
+            },
+            'kernel_ridge': {
+                'alpha': [0.1, 1, 10, 100],
+                'kernel': ['linear', 'poly', 'rbf', 'sigmoid'],
+                'gamma': [None, 0.1, 1, 10],
+                'degree': [2, 3, 4]
+            },
+            'pls': {
+                'n_components': [2, 5, 10, 20, 30],
+                'scale': [True, False],
+                'max_iter': [500, 1000, 2000]
+            },
+            'mlp': {
+                'hidden_layer_sizes': [(50,), (100,), (50, 50), (100, 50), (100, 100)],
+                'activation': ['relu', 'tanh', 'logistic'],
+                'alpha': [0.0001, 0.001, 0.01, 0.1],
+                'learning_rate': ['constant', 'invscaling', 'adaptive'],
+                'learning_rate_init': [0.001, 0.01, 0.1],
+                'max_iter': [200, 500, 1000],
+                'solver': ['adam', 'sgd', 'lbfgs'],
+                'batch_size': ['auto', 32, 64, 128],
+                'early_stopping': [True, False]
+            },
+            'knn': {
+                'n_neighbors': [3, 5, 7, 9, 11, 15, 21],
+                'weights': ['uniform', 'distance'],
+                'metric': ['euclidean', 'manhattan', 'minkowski', 'chebyshev'],
+                'p': [1, 2, 3],
+                'algorithm': ['auto', 'ball_tree', 'kd_tree', 'brute']
+            },
+            'radius_neighbors': {
+                'radius': [1.0, 2.0, 5.0, 10.0],
+                'weights': ['uniform', 'distance'],
+                'metric': ['euclidean', 'manhattan'],
+                'algorithm': ['auto', 'ball_tree', 'kd_tree', 'brute']
+            },
+            'decision_tree': {
+                'max_depth': [None, 5, 10, 20, 30, 50],
+                'min_samples_split': [2, 5, 10, 20, 30],
+                'min_samples_leaf': [1, 2, 4, 8, 16],
+                'max_features': ['sqrt', 'log2', None],
+                'splitter': ['best', 'random'],
+                'criterion': ['squared_error', 'friedman_mse', 'absolute_error']
+            },
+            'extra_trees': {
+                'n_estimators': [100, 200, 300, 500],
+                'max_depth': [10, 20, 30, None],
+                'min_samples_split': [2, 5, 10],
+                'min_samples_leaf': [1, 2, 4],
+                'max_features': ['sqrt', 'log2', None],
+                'bootstrap': [True, False]
+            },
+            'ada_boost': {
+                'n_estimators': [50, 100, 200, 300],
+                'learning_rate': [0.001, 0.01, 0.1, 0.5, 1.0],
+                'loss': ['linear', 'square', 'exponential'],
+                'estimator': [None]
+            },
+            'bagging': {
+                'n_estimators': [10, 50, 100],
+                'max_samples': [0.5, 0.7, 1.0],
+                'max_features': [0.5, 0.7, 1.0],
+                'bootstrap': [True, False],
+                'bootstrap_features': [True, False],
+                'estimator': [None]
+            },
+            'sgd': {
+                'loss': ['squared_error', 'huber', 'epsilon_insensitive', 'squared_epsilon_insensitive'],
+                'penalty': ['l2', 'l1', 'elasticnet', None],
+                'alpha': [0.0001, 0.001, 0.01, 0.1],
+                'learning_rate': ['constant', 'optimal', 'invscaling', 'adaptive'],
+                'eta0': [0.001, 0.01, 0.1],
+                'max_iter': [1000, 5000, 10000],
+                'early_stopping': [True, False]
+            },
+            'passive_aggressive': {
+                'C': [0.1, 1.0, 10.0],
+                'loss': ['epsilon_insensitive', 'squared_epsilon_insensitive'],
+                'max_iter': [1000, 5000],
+                'early_stopping': [True, False]
+            },
+            'ransac': {
+                'min_samples': [0.1, 0.5, 1.0],
+                'residual_threshold': [None, 1.0, 2.0, 5.0],
+                'max_trials': [100, 500, 1000],
+                'stop_probability': [0.99, 0.999],
+                'estimator': [None],
+                'loss': ['absolute_error', 'squared_error']
+            },
+            'theil_sen': {
+                'max_subpopulation': [10000, 50000, 100000],
+                'n_subsamples': [None, 100, 500, 1000],
+                'max_iter': [300, 500, 1000],
+                'random_state': [42]
+            },
+            'orthogonal_matching_pursuit': {
+                'n_nonzero_coefs': [None, 5, 10, 20, 50],
+                'tol': [None, 1e-3, 1e-2],
+                'fit_intercept': [True, False],
+                'precompute': ['auto', True, False]
+            },
+            'gaussian_process': {
+                'kernel': [None],
+                'alpha': [1e-10, 1e-5, 1e-2],
+                'optimizer': ['fmin_l_bfgs_b', None],
+                'n_restarts_optimizer': [0, 5, 10],
+                'normalize_y': [True, False]
+            },
+            'isotonic': {
+                'increasing': [True, False, 'auto'],
+                'out_of_bounds': ['nan', 'clip', 'raise']
+            }
+        }
+
+    def setup_scaler(self):
+        """
+        Configure the feature scaler.
+        """
+        scalers = {
+            'standard': StandardScaler,
+            'minmax': MinMaxScaler,
+            'robust': RobustScaler,
+            'quantile': QuantileTransformer,
+            'power': PowerTransformer
+        }
+
+        if self.scaler_type in scalers:
+            self.scaler_class = scalers[self.scaler_type]
+        else:
+            self.scaler_class = StandardScaler
+            logger.warning(f"Unknown scaler_type: {self.scaler_type}. Using StandardScaler.")
+
+    def setup_imputer(self):
+        """
+        Configure the missing value imputer.
+        """
+        imputers = {
+            'mean': SimpleImputer(strategy='mean'),
+            'median': SimpleImputer(strategy='median'),
+            'most_frequent': SimpleImputer(strategy='most_frequent'),
+            'constant': SimpleImputer(strategy='constant', fill_value=0),
+            'knn': KNNImputer(n_neighbors=5)
+        }
+
+        if self.impute_method in imputers:
+            self.imputer = imputers[self.impute_method]
+        else:
+            self.imputer = SimpleImputer(strategy='median')
+
+    def to_dict(self):
+        """
+        Convert configuration to a dictionary.
+        """
+        config_dict = asdict(self)
+        config_dict.pop('scaler_class', None)
+        config_dict.pop('imputer', None)
+        return config_dict
+
+    def save(self, path: str = None):
+        """
+        Save configuration to a JSON file.
+        """
+        if path is None:
+            path = f'{self.ml_results_dir}/ml_config.json'
+
+        with open(path, 'w', encoding='utf-8') as f:
+            json.dump(self.to_dict(), f, indent=4, ensure_ascii=False)
+
+        logger.info(f"Configuration saved: {path}")
+
+    @classmethod
+    def load(cls, path: str):
+        """
+        Load configuration from a JSON file.
+        """
+        with open(path, 'r', encoding='utf-8') as f:
+            config_dict = json.load(f)
+
+        config = cls(**config_dict)
+        config.setup_scaler()
+        config.setup_imputer()
+        return config

src/ml_pipeline/core/base_model.py

@@ -0,0 +1,909 @@
+import os
+from pathlib import Path
+import numpy as np
+import pandas as pd
+import hashlib
+import re
+import pickle
+import joblib
+import json
+import time
+import sys
+from datetime import datetime
+from typing import Dict, List, Optional, Any, Union
+from collections import defaultdict
+import traceback
+import logging
+
+import shap
+from sklearn.base import BaseEstimator, RegressorMixin
+from sklearn.model_selection import GroupKFold, KFold, LeaveOneOut, ShuffleSplit, StratifiedKFold, TimeSeriesSplit, cross_validate
+from sklearn.pipeline import Pipeline
+from sklearn.metrics import (
+    make_scorer, mean_squared_error, mean_absolute_error, mean_squared_log_error, r2_score,
+    explained_variance_score, median_absolute_error, max_error
+)
+from sklearn.inspection import permutation_importance
+from sklearn.preprocessing import PolynomialFeatures
+import scipy.stats as stats
+
+from src.ml_pipeline.utils.validators import SHAP_AVAILABLE
+
+from ..config.ml_config import MLConfig
+from .resource_monitor import ResourceMonitor
+
+logger = logging.getLogger(__name__)
+
+
+class BaseMLModel:
+    """Base class for ML models with extended functionality."""
+
+    def __init__(self, config: MLConfig, model_name: str):
+        self.config = config
+        self.model_name = model_name
+        self.model = None
+        self.pipeline = None
+        self.best_params = {}
+        self.cv_results = {}
+        self.train_metrics = {}
+        self.val_metrics = {}
+        self.test_metrics = {}
+        self.feature_importances = {}
+        self.shap_values = None
+        self.X_sample_shap = None
+        self.predictions = {}
+        self.residuals = {}
+        self.training_time = None
+        self.inference_time = None
+        self.is_fitted = False
+        self.model_hash = None
+        self.feature_names = None
+        self.resource_monitor = ResourceMonitor()
+        self.X_train_clean = None
+
+    def create_pipeline(self, **model_params):
+        """Create pipeline with preprocessing and model."""
+        steps = []
+
+        if self.config.impute_missing:
+            steps.append(('imputer', self.config.imputer))
+
+        if self.config.scale_features:
+            steps.append(('scaler', self.config.scaler_class()))
+
+        if self.config.feature_engineering and self.config.create_polynomial_features:
+            steps.append(('poly_features', PolynomialFeatures(
+                degree=self.config.polynomial_degree,
+                include_bias=False,
+                interaction_only=self.config.create_interaction_features
+            )))
+
+        steps.append((self.model_name, self.model))
+
+        self.pipeline = Pipeline(steps, verbose=self.config.verbose)
+        return self.pipeline
+
+    def clean_feature_names(self, X):
+        """Improved feature name cleaning for compatibility with LightGBM and other models."""
+        if isinstance(X, pd.DataFrame):
+            X_clean = X.copy()
+            new_columns = []
+
+            for col in X_clean.columns:
+                col_str = str(col)
+
+                # For LightGBM strict requirements: only letters, numbers, underscores
+                # Remove all special characters and spaces
+                col_clean = re.sub(r'[^a-zA-Z0-9_]', '_', col_str)
+
+                # Remove multiple underscores
+                col_clean = re.sub(r'_{2,}', '_', col_clean)
+
+                # Remove leading and trailing underscores
+                col_clean = col_clean.strip('_')
+
+                # If string is empty after cleaning
+                if not col_clean:
+                    col_clean = f'feature_{abs(hash(col_str)) % 10000:04d}'
+
+                # LightGBM requires names to start with a letter
+                if col_clean and col_clean[0].isdigit():
+                    col_clean = f'feature_{col_clean}'
+
+                # Check name is not empty and not too long
+                if not col_clean:
+                    col_clean = f'feature_{abs(hash(col_str)) % 10000:04d}'
+
+                # Limit name length (LightGBM may have restrictions)
+                if len(col_clean) > 100:
+                    col_clean = col_clean[:100]
+
+                # Check for duplicates
+                base_name = col_clean
+                counter = 1
+                original_name = col_clean
+                while col_clean in new_columns:
+                    col_clean = f'{original_name}_{counter}'
+                    counter += 1
+                    if counter > 100:  # Protection against infinite loop
+                        col_clean = f'{original_name}_{hash(col_str) % 10000:04d}'
+                        break
+
+                new_columns.append(col_clean)
+
+            X_clean.columns = new_columns
+            self.feature_names = list(X_clean.columns)
+
+            if self.config.debug:
+                logger.debug(f"Cleaned feature names. Example: {list(X.columns[:3])} -> {new_columns[:3]}")
+
+            return X_clean
+
+        return X
+
+    def clean_feature_names_for_lgbm(self, X):
+        """Special cleaning for LightGBM."""
+        if isinstance(X, pd.DataFrame):
+            X_clean = X.copy()
+            new_columns = []
+
+            for i, col in enumerate(X_clean.columns):
+                col_str = str(col)
+
+                # LightGBM is very strict - use only simple names
+                col_clean = re.sub(r'[^a-zA-Z0-9_]', '', col_str)  # Remove all non-alphanumeric
+
+                if not col_clean:
+                    col_clean = f'feature_{i:04d}'
+
+                # Limit length
+                col_clean = col_clean[:50]
+
+                # Check for duplicates
+                base_name = col_clean
+                counter = 1
+                while col_clean in new_columns:
+                    col_clean = f'{base_name}_{counter}'
+                    counter += 1
+
+                new_columns.append(col_clean)
+
+            X_clean.columns = new_columns
+            return X_clean
+
+        return X
+
+    def fit(self, X_train, y_train, X_val=None, y_val=None, **kwargs):
+        """Train model with resource monitoring."""
+        self.resource_monitor.start()
+
+        try:
+            start_time = datetime.now()
+
+            self.X_train_clean = self.clean_feature_names(X_train)
+
+            if self.config.use_pipeline:
+                if self.pipeline is None:
+                    self.create_pipeline()
+
+                self.resource_monitor.snapshot('pipeline_fit_start')
+                self.pipeline.fit(self.X_train_clean, y_train)
+                self.model = self.pipeline.named_steps[self.model_name]
+            else:
+                self.resource_monitor.snapshot('model_fit_start')
+                self.model.fit(self.X_train_clean, y_train)
+
+            self.training_time = (datetime.now() - start_time).total_seconds()
+            self.is_fitted = True
+
+            self.resource_monitor.snapshot('fit_complete')
+
+            self.model_hash = hashlib.md5(
+                f"{self.model_name}_{datetime.now().timestamp()}".encode()
+            ).hexdigest()[:8]
+
+            logger.info(f"✓ Model {self.model_name} trained in {self.training_time:.2f} seconds")
+
+            self.predictions['train'] = self.predict(X_train)
+            self.residuals['train'] = y_train - self.predictions['train']
+
+            if X_val is not None and y_val is not None:
+                self.predictions['val'] = self.predict(X_val)
+                self.residuals['val'] = y_val - self.predictions['val']
+
+            return self
+
+        except Exception as e:
+            logger.error(f"✗ Error training model {self.model_name}: {e}")
+            logger.error(traceback.format_exc())
+            raise
+
+    def predict(self, X):
+        """Make predictions with timing measurement."""
+        if not self.is_fitted:
+            raise ValueError("Model is not trained")
+
+        start_time = datetime.now()
+
+        try:
+            X_clean = self.clean_feature_names(X)
+
+            if self.config.use_pipeline and self.pipeline is not None:
+                predictions = self.pipeline.predict(X_clean)
+            else:
+                predictions = self.model.predict(X_clean)
+
+            if len(X) > 0:
+                self.inference_time = (datetime.now() - start_time).total_seconds() / len(X) * 1000
+            else:
+                self.inference_time = 0
+
+            return predictions
+        except Exception as e:
+            logger.error(f"Prediction error {self.model_name}: {e}")
+            raise
+
+    def evaluate(self, X, y, dataset_name: str = 'test'):
+        """Extended model evaluation."""
+        if not self.is_fitted:
+            raise ValueError("Model is not trained")
+
+        y_pred = self.predict(X)
+        metrics = self._calculate_metrics(y, y_pred, X)
+
+        if dataset_name == 'train':
+            self.train_metrics = metrics
+        elif dataset_name == 'val':
+            self.val_metrics = metrics
+        else:
+            self.test_metrics = metrics
+
+        self.predictions[dataset_name] = y_pred
+        self.residuals[dataset_name] = y - y_pred
+
+        return metrics
+
+    def _calculate_metrics(self, y_true, y_pred, X=None):
+        """Calculate extended metrics."""
+        metrics = {}
+
+        try:
+            residuals = y_true - y_pred
+
+            metrics['mse'] = mean_squared_error(y_true, y_pred)
+            metrics['rmse'] = np.sqrt(metrics['mse'])
+            metrics['mae'] = mean_absolute_error(y_true, y_pred)
+            metrics['r2'] = r2_score(y_true, y_pred)
+            metrics['explained_variance'] = explained_variance_score(y_true, y_pred)
+            metrics['median_absolute_error'] = median_absolute_error(y_true, y_pred)
+            metrics['max_error'] = max_error(y_true, y_pred)
+
+            mask = y_true != 0
+            if mask.any():
+                metrics['mape'] = np.mean(np.abs(residuals[mask] / y_true[mask])) * 100
+            else:
+                metrics['mape'] = np.nan
+
+            if (y_true >= 0).all() and (y_pred >= 0).all():
+                try:
+                    y_true_log = np.log1p(y_true)
+                    y_pred_log = np.log1p(y_pred)
+                    metrics['msle'] = mean_squared_log_error(y_true, y_pred)
+                    metrics['rmsle'] = np.sqrt(metrics['msle'])
+                except:
+                    metrics['msle'] = np.nan
+                    metrics['rmsle'] = np.nan
+            else:
+                metrics['msle'] = np.nan
+                metrics['rmsle'] = np.nan
+
+            metrics['wmape'] = np.sum(np.abs(residuals)) / np.sum(np.abs(y_true)) * 100 if np.sum(np.abs(y_true)) > 0 else np.nan
+
+            metrics['residual_mean'] = np.mean(residuals)
+            metrics['residual_std'] = np.std(residuals)
+            metrics['residual_skewness'] = stats.skew(residuals) if len(residuals) > 2 else np.nan
+            metrics['residual_kurtosis'] = stats.kurtosis(residuals) if len(residuals) > 3 else np.nan
+            metrics['residual_min'] = np.min(residuals)
+            metrics['residual_max'] = np.max(residuals)
+            metrics['residual_q1'] = np.percentile(residuals, 25)
+            metrics['residual_q3'] = np.percentile(residuals, 75)
+            metrics['residual_iqr'] = metrics['residual_q3'] - metrics['residual_q1']
+
+            std_dev = metrics['residual_std']
+            if not np.isnan(std_dev) and std_dev > 0:
+                for n in [1, 2, 3]:
+                    within_n_std = np.sum(np.abs(residuals) <= n * std_dev) / len(residuals) * 100
+                    metrics[f'within_{n}_std'] = within_n_std
+
+            if len(y_true) > 1:
+                metrics['correlation'] = np.corrcoef(y_true, y_pred)[0, 1]
+            else:
+                metrics['correlation'] = np.nan
+
+            metrics['bias'] = np.mean(y_pred - y_true)
+            metrics['mpe'] = np.mean((y_pred - y_true) / y_true) * 100 if np.all(y_true != 0) else np.nan
+
+            n = len(y_true)
+            if self.feature_names is not None:
+                p = len(self.feature_names)
+            else:
+                p = 1
+            metrics['adjusted_r2'] = 1 - (1 - metrics['r2']) * (n - 1) / (n - p - 1) if n > p + 1 and metrics['r2'] <= 1 else np.nan
+
+        except Exception as e:
+            logger.warning(f"Error calculating metrics: {e}")
+            if self.config.debug:
+                traceback.print_exc()
+
+        return metrics
+
+    def cross_validate(self, X, y, cv=None, scoring=None):
+        """Extended cross-validation."""
+        if scoring is None:
+            scoring = {
+                'neg_mse': 'neg_mean_squared_error',
+                'neg_rmse': make_scorer(lambda y_true, y_pred: -np.sqrt(mean_squared_error(y_true, y_pred))),
+                'neg_mae': 'neg_mean_absolute_error',
+                'r2': 'r2',
+                'explained_variance': 'explained_variance'
+            }
+
+        if cv is None:
+            if self.config.cv_method == 'timeseries':
+                cv = TimeSeriesSplit(n_splits=min(self.config.cv_folds, len(X) // 2))
+            elif self.config.cv_method == 'stratified':
+                if len(y) > 0:
+                    y_bins = pd.qcut(y, q=min(5, len(y) // 10), labels=False, duplicates='drop')
+                    cv = StratifiedKFold(n_splits=self.config.cv_folds, shuffle=True,
+                                        random_state=self.config.random_state)
+                else:
+                    cv = self.config.cv_folds
+            elif self.config.cv_method == 'group':
+                if self.config.groups is not None:
+                    groups = X[self.config.groups] if self.config.groups in X.columns else None
+                    cv = GroupKFold(n_splits=self.config.cv_folds)
+                else:
+                    cv = self.config.cv_folds
+            elif self.config.cv_method == 'loo':
+                cv = LeaveOneOut()
+            elif self.config.cv_method == 'shuffle':
+                cv = ShuffleSplit(n_splits=self.config.cv_folds, test_size=0.2,
+                                 random_state=self.config.random_state)
+            else:
+                cv = KFold(n_splits=self.config.cv_folds, shuffle=True,
+                          random_state=self.config.random_state)
+
+        try:
+            X_clean = self.clean_feature_names(X)
+
+            if self.config.use_pipeline and self.pipeline is not None:
+                estimator = self.pipeline
+            else:
+                estimator = self.model
+
+            cv_results = cross_validate(
+                estimator, X_clean, y,
+                cv=cv,
+                scoring=scoring,
+                n_jobs=min(2, self.config.n_jobs),
+                return_train_score=True,
+                error_score='raise',
+                return_estimator=True
+            )
+
+            self.cv_results = {
+                'train_neg_mse_mean': -np.mean(cv_results['train_neg_mse']),
+                'train_neg_mse_std': np.std(cv_results['train_neg_mse']),
+                'test_neg_mse_mean': -np.mean(cv_results['test_neg_mse']),
+                'test_neg_mse_std': np.std(cv_results['test_neg_mse']),
+                'train_r2_mean': np.mean(cv_results['train_r2']),
+                'train_r2_std': np.std(cv_results['train_r2']),
+                'test_r2_mean': np.mean(cv_results['test_r2']),
+                'test_r2_std': np.std(cv_results['test_r2']),
+                'fit_time_mean': np.mean(cv_results['fit_time']),
+                'score_time_mean': np.mean(cv_results['score_time']),
+                'train_explained_variance_mean': np.mean(cv_results['train_explained_variance']),
+                'test_explained_variance_mean': np.mean(cv_results['test_explained_variance']),
+                'estimators': cv_results['estimator']
+            }
+
+            return self.cv_results
+
+        except Exception as e:
+            logger.warning(f"Cross-validation failed: {e}")
+            return {}
+
+    def get_feature_importance(self, feature_names=None, method='auto'):
+        """Get feature importance using different methods."""
+        if not self.is_fitted:
+            return {}
+
+        if feature_names is None:
+            feature_names = self.feature_names
+
+        if feature_names is None:
+            return {}
+
+        importances = {}
+
+        try:
+            if method in ['auto', 'native']:
+                if hasattr(self.model, 'feature_importances_'):
+                    importances = self.model.feature_importances_
+                elif hasattr(self.model, 'coef_'):
+                    importances = np.abs(self.model.coef_)
+                    if len(importances.shape) > 1:
+                        importances = np.mean(importances, axis=0)
+                elif hasattr(self.model, 'estimators_'):
+                    importances = np.mean([
+                        tree.feature_importances_ for tree in self.model.estimators_
+                    ], axis=0)
+
+            if method in ['auto', 'permutation'] and len(feature_names) <= 50:
+                try:
+                    if self.X_train_clean is not None:
+                        perm_importance = permutation_importance(
+                            self.model, self.X_train_clean,
+                            np.random.randn(len(self.X_train_clean)),
+                            n_repeats=10,
+                            random_state=self.config.random_state,
+                            n_jobs=min(2, self.config.n_jobs)
+                        )
+                        importances_perm = perm_importance.importances_mean
+
+                        if method == 'permutation':
+                            importances = importances_perm
+                        else:
+                            if len(importances) == 0:
+                                importances = importances_perm
+                except:
+                    pass
+
+            if len(importances) > 0 and len(importances) == len(feature_names):
+                self.feature_importances = dict(zip(feature_names, importances))
+                self.feature_importances = dict(sorted(
+                    self.feature_importances.items(),
+                    key=lambda x: x[1],
+                    reverse=True
+                ))
+
+            return self.feature_importances
+
+        except Exception as e:
+            logger.debug(f"Feature importance not available for {self.model_name}: {e}")
+            return {}
+
+    def calculate_shap_values(self, X, sample_size=1000):
+        """Calculate SHAP values for model explanation with fixes."""
+        if not SHAP_AVAILABLE or not self.is_fitted:
+            return None
+
+        try:
+            X_clean = self.clean_feature_names(X)
+
+            if sample_size < len(X_clean):
+                indices = np.random.choice(len(X_clean), size=min(sample_size, len(X_clean)), replace=False)
+                X_sample = X_clean.iloc[indices]
+            else:
+                X_sample = X_clean
+
+            if len(X_sample) == 0:
+                return None
+
+            # Save sample for visualisation
+            self.X_sample_shap = X_sample
+
+            # Use different explainers for different model types
+            if hasattr(self.model, 'feature_importances_') or hasattr(self.model, 'feature_importance'):
+                # Tree-based models
+                try:
+                    explainer = shap.TreeExplainer(self.model)
+                    self.shap_values = explainer.shap_values(X_sample)
+                except:
+                    # Alternative approach for complex cases
+                    try:
+                        explainer = shap.Explainer(self.model, X_sample)
+                        self.shap_values = explainer(X_sample)
+                    except Exception as e:
+                        logger.warning(f"Failed to use TreeExplainer: {e}")
+                        return None
+            elif hasattr(self.model, 'coef_'):
+                # Linear models
+                try:
+                    explainer = shap.LinearExplainer(self.model, X_sample)
+                    self.shap_values = explainer.shap_values(X_sample)
+                except:
+                    explainer = shap.KernelExplainer(self.model.predict, X_sample)
+                    self.shap_values = explainer.shap_values(X_sample)
+            else:
+                # Other models
+                try:
+                    explainer = shap.KernelExplainer(self.model.predict, X_sample)
+                    self.shap_values = explainer.shap_values(X_sample)
+                except Exception as e:
+                    logger.warning(f"Failed to use KernelExplainer: {e}")
+                    return None
+
+            return self.shap_values
+
+        except Exception as e:
+            logger.warning(f"Failed to calculate SHAP values: {e}")
+            if self.config.debug:
+                logger.debug(traceback.format_exc())
+            return None
+
+    def save_model(self, path: str = None, max_retries: int = 3):
+        """
+        Reliable model saving that works for all model types.
+        Returns path to saved file or None on error.
+        """
+        if not self.is_fitted:
+            logger.warning(f"Model {self.model_name} is not trained, cannot save")
+            return None
+
+        # Create unique filename
+        if path is None:
+            timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
+            filename = f'{self.model_name}_{self.model_hash}_{timestamp}'
+            
+            # Determine base path
+            if 'google.colab' in str(sys.modules):
+                base_path = '/content/drive/MyDrive/saved_models'
+            else:
+                base_path = self.config.models_dir if hasattr(self.config, 'models_dir') else 'saved_models'
+                
+            Path(base_path).mkdir(parents=True, exist_ok=True)
+            path = f'{base_path}/{filename}'
+        
+        try:
+            # Create directory if it doesn't exist
+            Path(os.path.dirname(path)).mkdir(parents=True, exist_ok=True)
+            
+            # 1. Determine model type and use appropriate saving method
+            model_type = type(self.model).__name__.lower()
+            
+            # 2. Use different saving strategies for different model types
+            saved_files = []
+            
+            for attempt in range(max_retries):
+                try:
+                    # STRATEGY 1: Models with built-in save_model() method
+                    if hasattr(self.model, 'save_model') and callable(getattr(self.model, 'save_model')):
+                        if 'lightgbm' in model_type:
+                            # LightGBM
+                            model_path = f"{path}_lightgbm.txt"
+                            self.model.save_model(model_path)
+                            saved_files.append(('lightgbm_model', model_path))
+                            
+                        elif 'catboost' in model_type:
+                            # CatBoost
+                            model_path = f"{path}_catboost.cbm"
+                            self.model.save_model(model_path, format='cbm')
+                            saved_files.append(('catboost_model', model_path))
+                            
+                        elif 'xgboost' in model_type:
+                            # XGBoost
+                            model_path = f"{path}_xgboost.json"
+                            self.model.save_model(model_path)
+                            saved_files.append(('xgboost_model', model_path))
+                            
+                        else:
+                            # Other models with save_model (Prophet etc.)
+                            model_path = f"{path}_native.model"
+                            self.model.save_model(model_path)
+                            saved_files.append(('native_model', model_path))
+                    
+                    # STRATEGY 2: Sklearn and other Python objects
+                    else:
+                        # For sklearn models and pipelines
+                        model_path = f"{path}_sklearn.pkl"
+                        
+                        # Remove problematic attributes before saving
+                        model_to_save = self.model
+                        
+                        # For pipelines, remove data references
+                        if hasattr(model_to_save, 'named_steps'):
+                            # This is a pipeline, clone it without data
+                            from sklearn.base import clone
+                            model_to_save = clone(model_to_save)
+                        
+                        # Save model
+                        joblib.dump(model_to_save, model_path, compress=3, protocol=pickle.HIGHEST_PROTOCOL)
+                        saved_files.append(('sklearn_model', model_path))
+                    
+                    # 3. Save metadata SEPARATELY (this is key)
+                    metadata = {
+                        'model_name': self.model_name,
+                        'model_type': str(type(self.model)),
+                        'model_hash': self.model_hash,
+                        'best_params': self._convert_to_serializable(self.best_params),
+                        'train_metrics': self._convert_to_serializable(self.train_metrics),
+                        'test_metrics': self._convert_to_serializable(self.test_metrics),
+                        'training_time': self.training_time,
+                        'inference_time': self.inference_time,
+                        'feature_names': self.feature_names,
+                        'config': self.config.to_dict() if hasattr(self.config, 'to_dict') else str(self.config),
+                        'saved_files': saved_files,
+                        'python_version': sys.version,
+                        'library_versions': self._get_library_versions(),
+                        'saved_at': datetime.now().isoformat(),
+                        'model_version': '2.0'
+                    }
+                    
+                    # Feature importance (if available, save separately)
+                    if self.feature_importances:
+                        importance_path = f"{path}_feature_importance.json"
+                        with open(importance_path, 'w', encoding='utf-8') as f:
+                            json.dump(self._convert_to_serializable(self.feature_importances), 
+                                     f, indent=4, ensure_ascii=False)
+                        saved_files.append(('feature_importance', importance_path))
+                    
+                    # Save metadata
+                    metadata_path = f"{path}_metadata.json"
+                    with open(metadata_path, 'w', encoding='utf-8') as f:
+                        json.dump(metadata, f, indent=4, default=str, ensure_ascii=False)
+                    saved_files.append(('metadata', metadata_path))
+                    
+                    # 4. Create file manifest
+                    manifest = {
+                        'model_name': self.model_name,
+                        'main_model_file': saved_files[0][1] if saved_files else None,
+                        'all_files': [file_info for _, file_info in saved_files],
+                        'load_instructions': self._get_load_instructions(saved_files)
+                    }
+                    
+                    manifest_path = f"{path}_manifest.json"
+                    with open(manifest_path, 'w', encoding='utf-8') as f:
+                        json.dump(manifest, f, indent=4, ensure_ascii=False)
+                    
+                    logger.info(f"✓ Model {self.model_name} successfully saved")
+                    logger.info(f"  Main file: {saved_files[0][1] if saved_files else 'N/A'}")
+                    logger.info(f"  Total files: {len(saved_files)}")
+                    
+                    return manifest_path
+                    
+                except (pickle.PicklingError, TypeError) as e:
+                    logger.warning(f"Attempt {attempt + 1}/{max_retries}: Serialisation error: {str(e)[:100]}")
+                    
+                    if attempt == max_retries - 1:  # Last attempt
+                        # Try saving only parameters
+                        return self._save_model_params_only(path)
+                        
+                    time.sleep(1)  # Wait before retry
+                    
+                except Exception as e:
+                    logger.error(f"Attempt {attempt + 1}/{max_retries}: Error: {e}")
+                    if attempt == max_retries - 1:
+                        return self._save_model_params_only(path)
+                    time.sleep(1)
+        
+        except Exception as e:
+            logger.error(f"Critical error saving model {self.model_name}: {e}")
+            return None
+
+    def _convert_to_serializable(self, obj):
+        """Recursively convert object to serializable format."""
+        if obj is None:
+            return None
+        elif isinstance(obj, (str, int, float, bool)):
+            return obj
+        elif isinstance(obj, dict):
+            return {k: self._convert_to_serializable(v) for k, v in obj.items()}
+        elif isinstance(obj, (list, tuple, set)):
+            return [self._convert_to_serializable(item) for item in obj]
+        elif isinstance(obj, np.integer):
+            return int(obj)
+        elif isinstance(obj, np.floating):
+            return float(obj)
+        elif isinstance(obj, np.ndarray):
+            return obj.tolist()
+        elif isinstance(obj, pd.Series):
+            return obj.to_dict()
+        elif isinstance(obj, pd.DataFrame):
+            return obj.to_dict('records')
+        elif hasattr(obj, 'tolist'):
+            return obj.tolist()
+        else:
+            return str(obj)
+
+    def _save_model_params_only(self, base_path):
+        """Emergency save of model parameters only."""
+        try:
+            params_path = f"{base_path}_params_only.json"
+            
+            # Try to get model parameters
+            model_params = {}
+            if hasattr(self.model, 'get_params'):
+                model_params = self.model.get_params()
+            
+            # Save minimal information
+            minimal_data = {
+                'model_name': self.model_name,
+                'model_type': str(type(self.model)),
+                'model_params': self._convert_to_serializable(model_params),
+                'train_metrics': self._convert_to_serializable(self.train_metrics),
+                'test_metrics': self._convert_to_serializable(self.test_metrics),
+                'feature_names': self.feature_names,
+                'saved_at': datetime.now().isoformat(),
+                'note': 'Model parameters only (full model could not be saved)'
+            }
+            
+            with open(params_path, 'w', encoding='utf-8') as f:
+                json.dump(minimal_data, f, indent=4, ensure_ascii=False)
+            
+            logger.warning(f"⚠ Only model parameters saved: {params_path}")
+            return params_path
+            
+        except Exception as e:
+            logger.error(f"Failed to save even model parameters: {e}")
+            return None
+
+    def _get_library_versions(self):
+        """Get versions of used libraries."""
+        versions = {}
+        libs = ['sklearn', 'numpy', 'pandas', 'xgboost', 'lightgbm', 'catboost']
+        
+        for lib in libs:
+            try:
+                module = __import__(lib)
+                versions[lib] = getattr(module, '__version__', 'unknown')
+            except:
+                pass
+        
+        return versions
+
+    def _get_load_instructions(self, saved_files):
+        """Generate loading instructions for this model type."""
+        instructions = {
+            'sklearn.pkl': 'joblib.load(file_path)',
+            'lightgbm.txt': 'lightgbm.Booster(model_file=file_path)',
+            'catboost.cbm': 'catboost.CatBoost().load_model(file_path)',
+            'xgboost.json': 'xgboost.XGBRegressor().load_model(file_path)',
+            'metadata.json': 'json.load(open(file_path))',
+            'params_only.json': 'Parameters only, full model cannot be restored'
+        }
+        
+        return instructions
+    
+    def load_model(self, manifest_path: str):
+        """Universal model loading from saved files."""
+        try:
+            # Load manifest
+            with open(manifest_path, 'r', encoding='utf-8') as f:
+                manifest = json.load(f)
+            
+            # Load metadata
+            metadata_path = None
+            model_file = None
+            
+            for file_info in manifest.get('all_files', []):
+                if 'metadata' in file_info or file_info.endswith('_metadata.json'):
+                    metadata_path = file_info
+                elif 'sklearn.pkl' in file_info or 'lightgbm.txt' in file_info or 'catboost.cbm' in file_info or 'xgboost.json' in file_info:
+                    model_file = file_info
+            
+            if not metadata_path:
+                raise ValueError("Metadata file not found")
+            
+            # Load metadata
+            with open(metadata_path, 'r', encoding='utf-8') as f:
+                metadata = json.load(f)
+            
+            # Restore attributes
+            self.model_name = metadata['model_name']
+            self.model_hash = metadata.get('model_hash', 'unknown')
+            self.best_params = metadata.get('best_params', {})
+            self.train_metrics = metadata.get('train_metrics', {})
+            self.test_metrics = metadata.get('test_metrics', {})
+            self.training_time = metadata.get('training_time', 0)
+            self.inference_time = metadata.get('inference_time', 0)
+            self.feature_names = metadata.get('feature_names', [])
+            self.feature_importances = {}
+            
+            # Load model
+            if model_file:
+                if model_file.endswith('.pkl'):
+                    # sklearn model
+                    self.model = joblib.load(model_file)
+                elif model_file.endswith('.txt') and 'lightgbm' in str(metadata.get('model_type', '')):
+                    # LightGBM
+                    import lightgbm as lgb
+                    self.model = lgb.Booster(model_file=model_file)
+                elif model_file.endswith('.cbm'):
+                    # CatBoost
+                    import catboost as cb
+                    self.model = cb.CatBoostRegressor()
+                    self.model.load_model(model_file)
+                elif model_file.endswith('.json') and 'xgboost' in str(metadata.get('model_type', '')):
+                    # XGBoost
+                    import xgboost as xgb
+                    self.model = xgb.XGBRegressor()
+                    self.model.load_model(model_file)
+                else:
+                    raise ValueError(f"Unknown model format: {model_file}")
+            
+            # Load feature importance if available
+            for file_info in manifest.get('all_files', []):
+                if 'feature_importance' in file_info or file_info.endswith('_feature_importance.json'):
+                    with open(file_info, 'r', encoding='utf-8') as f:
+                        self.feature_importances = json.load(f)
+                    break
+            
+            self.is_fitted = True
+            logger.info(f"✓ Model {self.model_name} successfully loaded")
+            return self
+            
+        except Exception as e:
+            logger.error(f"Error loading model: {e}")
+            raise
+            
+    def get_summary(self):
+        """Get extended model summary."""
+        summary = {
+            'model_name': self.model_name,
+            'is_fitted': self.is_fitted,
+            'training_time': self.training_time,
+            'inference_time': self.inference_time,
+            'best_params': self.best_params,
+            'train_metrics': self.train_metrics,
+            'test_metrics': self.test_metrics,
+            'cv_results': {k: v for k, v in self.cv_results.items() if k != 'estimators'},
+            'feature_importance_count': len(self.feature_importances),
+            'model_hash': self.model_hash,
+            'has_shap_values': self.shap_values is not None,
+            'resource_usage': self.resource_monitor.get_summary()
+        }
+
+        return summary
+
+    def get_learning_curve(self, X_train, y_train, train_sizes=None):
+        """Get learning curve."""
+        from sklearn.model_selection import learning_curve
+
+        if train_sizes is None:
+            train_sizes = np.linspace(0.1, 1.0, 10)
+
+        try:
+            if self.config.use_pipeline and self.pipeline is not None:
+                estimator = self.pipeline
+            else:
+                estimator = self.model
+
+            X_clean = self.clean_feature_names(X_train)
+
+            train_sizes_abs, train_scores, test_scores = learning_curve(
+                estimator, X_clean, y_train,
+                train_sizes=train_sizes,
+                cv=min(5, self.config.cv_folds),
+                scoring='neg_mean_squared_error',
+                n_jobs=min(2, self.config.n_jobs),
+                shuffle=True,
+                random_state=self.config.random_state,
+                error_score='raise'
+            )
+
+            return {
+                'train_sizes': train_sizes_abs,
+                'train_scores': train_scores,
+                'test_scores': test_scores
+            }
+        except Exception as e:
+            logger.warning(f"Failed to build learning curve: {e}")
+            return None
+
+    def get_sklearn_estimator(self):
+        """Return sklearn-compatible model or pipeline."""
+        if not self.is_fitted:
+            raise ValueError("Model is not trained")
+
+        if self.config.use_pipeline and self.pipeline is not None:
+            return self.pipeline
+        else:
+            return self.model
+
+    def get_sklearn_model(self):
+        """Return model without pipeline."""
+        if not self.is_fitted:
+            raise ValueError("Model is not trained")
+
+        return self.model

src/ml_pipeline/core/model_factory.py

@@ -0,0 +1,124 @@
+import sys
+import os
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from src.ml_pipeline.models.neighbors_models import KNNModel, RadiusNeighborsModel
+from src.ml_pipeline.models.neural_models import GaussianProcessModel, MLPModel
+from src.ml_pipeline.models.svm_models import LinearSVMModel, NuSVMModel, SVMModel
+
+import logging
+from typing import Dict, List
+from ..config.ml_config import MLConfig
+from .base_model import BaseMLModel
+from ..models.linear_models import *
+from ..models.tree_models import *
+from ..models.boosting_models import *
+from ..models.ensemble_models import *
+
+logger = logging.getLogger(__name__)
+
+
+class ModelFactory:
+    """Factory for creating ML model instances."""
+
+    @staticmethod
+    def create_model(model_type: str, config: MLConfig) -> BaseMLModel:
+        model_registry = {
+            # Linear models
+            'linear_regression': LinearRegressionModel,
+            'ridge': RidgeRegressionModel,
+            'lasso': LassoRegressionModel,
+            'elasticnet': ElasticNetModel,
+            'bayesian_ridge': BayesianRidgeModel,
+            'huber': HuberRegressionModel,
+            'kernel_ridge': KernelRidgeModel,
+            'pls': PLSRegressionModel,
+            'sgd': SGDRegressionModel,
+            'passive_aggressive': PassiveAggressiveRegressionModel,
+            'ransac': RANSACRegressionModel,
+            'theil_sen': TheilSenRegressionModel,
+            'orthogonal_matching_pursuit': OrthogonalMatchingPursuitModel,
+            'isotonic': IsotonicRegressionModel,
+
+            # Tree-based models
+            'random_forest': RandomForestModel,
+            'extra_trees': ExtraTreesModel,
+            'decision_tree': DecisionTreeModel,
+
+            # Gradient boosting models
+            'gradient_boosting': GradientBoostingModel,
+            'hist_gradient_boosting': HistGradientBoostingModel,
+            'xgboost': XGBoostModel,
+            'lightgbm': LightGBMModel,
+            'catboost': CatBoostModel,
+
+            # Ensemble models
+            'ada_boost': AdaBoostModel,
+            'bagging': BaggingModel,
+
+            # SVM models
+            'svr': SVMModel,
+            'linear_svr': LinearSVMModel,
+            'nu_svr': NuSVMModel,
+
+            # Nearest neighbours models
+            'knn': KNNModel,
+            'radius_neighbors': RadiusNeighborsModel,
+
+            # Neural network models
+            'mlp': MLPModel,
+            'gaussian_process': GaussianProcessModel
+        }
+
+        if model_type not in model_registry:
+            raise ValueError(f"Unknown model type: {model_type}")
+
+        return model_registry[model_type](config)
+
+    @staticmethod
+    def get_sklearn_compatible_estimator(model_wrapper: BaseMLModel):
+        """Extract a scikit-learn compatible estimator from the model wrapper."""
+        try:
+            if hasattr(model_wrapper, 'get_sklearn_estimator'):
+                return model_wrapper.get_sklearn_estimator()
+            elif hasattr(model_wrapper, 'pipeline') and model_wrapper.pipeline is not None:
+                return model_wrapper.pipeline
+            elif hasattr(model_wrapper, 'model') and model_wrapper.model is not None:
+                return model_wrapper.model
+            else:
+                raise ValueError("Unable to retrieve a scikit-learn compatible estimator.")
+        except Exception as e:
+            logger.error(f"Error retrieving scikit-learn compatible estimator: {e}")
+            raise
+
+    @staticmethod
+    def get_all_models(config: MLConfig, model_types: List[str] = None) -> Dict[str, BaseMLModel]:
+        if model_types is None:
+            model_types = config.baseline_models
+
+        models = {}
+        for model_type in model_types:
+            try:
+                models[model_type] = ModelFactory.create_model(model_type, config)
+                logger.info(f"✓ Model created: {model_type}")
+            except Exception as e:
+                logger.warning(f"✗ Failed to create model {model_type}: {e}")
+
+        return models
+
+    @staticmethod
+    def get_model_categories() -> Dict[str, List[str]]:
+        return {
+            'linear': ['linear_regression', 'ridge', 'lasso', 'elasticnet',
+                      'bayesian_ridge', 'huber', 'kernel_ridge', 'pls',
+                      'sgd', 'passive_aggressive', 'ransac', 'theil_sen',
+                      'orthogonal_matching_pursuit', 'isotonic'],
+            'tree': ['random_forest', 'extra_trees', 'decision_tree'],
+            'boosting': ['gradient_boosting', 'hist_gradient_boosting',
+                        'xgboost', 'lightgbm', 'catboost', 'ada_boost'],
+            'svm': ['svr', 'linear_svr', 'nu_svr'],
+            'neighbors': ['knn', 'radius_neighbors'],
+            'neural': ['mlp', 'gaussian_process'],
+            'ensemble': ['bagging']
+        }

src/ml_pipeline/core/resource_monitor.py

@@ -0,0 +1,150 @@
+import time
+import psutil
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from collections import defaultdict
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+class ResourceMonitor:
+    """Monitors system resource usage (CPU, memory, execution time)."""
+
+    def __init__(self):
+        self.start_time = None
+        self.start_memory = None
+        self.process = psutil.Process()
+        self.metrics = defaultdict(list)
+
+    def start(self):
+        """Start monitoring resources."""
+        self.start_time = time.time()
+        self.start_memory = self.process.memory_info().rss / 1024 ** 2
+        self.metrics.clear()
+        return self
+
+    def snapshot(self, label: str = None):
+        """Take a snapshot of current resource usage."""
+        current_time = time.time()
+        current_memory = self.process.memory_info().rss / 1024 ** 2
+
+        snapshot = {
+            'time': current_time - self.start_time,
+            'memory_mb': current_memory,
+            'memory_delta_mb': current_memory - self.start_memory,
+            'cpu_percent': self.process.cpu_percent(),
+            'threads': self.process.num_threads()
+        }
+
+        if label:
+            self.metrics[label].append(snapshot)
+
+        return snapshot
+
+    def get_summary(self):
+        """Generate a summary report of collected metrics."""
+        if not self.metrics:
+            return {}
+
+        summary = {}
+        for label, snapshots in self.metrics.items():
+            if snapshots:
+                times = [s['time'] for s in snapshots]
+                memories = [s['memory_mb'] for s in snapshots]
+
+                summary[label] = {
+                    'duration_sec': max(times) - min(times),
+                    'avg_memory_mb': np.mean(memories),
+                    'max_memory_mb': np.max(memories),
+                    'min_memory_mb': np.min(memories),
+                    'memory_peak_mb': max(memories),
+                    'snapshots': len(snapshots)
+                }
+
+        return summary
+
+    def plot_usage(self, save_path: str = None):
+        """Plot resource usage metrics."""
+        if not self.metrics:
+            logger.warning("No data available for plotting.")
+            return
+
+        fig, axes = plt.subplots(2, 2, figsize=(15, 10))
+
+        all_labels = []
+        all_times = []
+        all_memories = []
+        all_cpus = []
+
+        for label, snapshots in self.metrics.items():
+            for snapshot in snapshots:
+                all_labels.append(label)
+                all_times.append(snapshot['time'])
+                all_memories.append(snapshot['memory_mb'])
+                all_cpus.append(snapshot['cpu_percent'])
+
+        if not all_times:
+            return
+
+        df = pd.DataFrame({
+            'label': all_labels,
+            'time': all_times,
+            'memory': all_memories,
+            'cpu': all_cpus
+        })
+
+        # Plot memory usage over time
+        for label in df['label'].unique():
+            label_data = df[df['label'] == label]
+            axes[0, 0].plot(label_data['time'], label_data['memory'],
+                          'o-', label=label, alpha=0.7)
+
+        axes[0, 0].set_xlabel('Time (sec)')
+        axes[0, 0].set_ylabel('Memory (MB)')
+        axes[0, 0].set_title('Memory Usage Over Time')
+        axes[0, 0].legend(loc='best')
+        axes[0, 0].grid(True, alpha=0.3)
+
+        # Plot CPU usage over time
+        for label in df['label'].unique():
+            label_data = df[df['label'] == label]
+            axes[0, 1].plot(label_data['time'], label_data['cpu'],
+                          's-', label=label, alpha=0.7)
+
+        axes[0, 1].set_xlabel('Time (sec)')
+        axes[0, 1].set_ylabel('CPU (%)')
+        axes[0, 1].set_title('CPU Usage Over Time')
+        axes[0, 1].legend(loc='best')
+        axes[0, 1].grid(True, alpha=0.3)
+
+        # Plot peak memory per operation
+        peak_memory = df.groupby('label')['memory'].max().sort_values()
+        axes[1, 0].bar(range(len(peak_memory)), peak_memory.values,
+                      color='skyblue', edgecolor='black')
+        axes[1, 0].set_xticks(range(len(peak_memory)))
+        axes[1, 0].set_xticklabels(peak_memory.index, rotation=45, ha='right')
+        axes[1, 0].set_xlabel('Operation')
+        axes[1, 0].set_ylabel('Peak Memory (MB)')
+        axes[1, 0].set_title('Peak Memory Usage by Operation')
+        axes[1, 0].grid(True, alpha=0.3, axis='y')
+
+        # Plot memory distribution boxplot
+        memory_data = [df[df['label'] == label]['memory'].values
+                      for label in df['label'].unique()]
+        axes[1, 1].boxplot(memory_data, labels=df['label'].unique())
+        axes[1, 1].set_xlabel('Operation')
+        axes[1, 1].set_ylabel('Memory (MB)')
+        axes[1, 1].set_title('Memory Usage Distribution')
+        axes[1, 1].grid(True, alpha=0.3, axis='y')
+        plt.setp(axes[1, 1].xaxis.get_majorticklabels(), rotation=45, ha='right')
+
+        plt.tight_layout()
+
+        if save_path:
+            plt.savefig(save_path, dpi=300, bbox_inches='tight')
+            logger.info(f"Resource usage plot saved: {save_path}")
+
+        plt.show()
+        plt.close(fig)

src/ml_pipeline/features/engineering.py

@@ -0,0 +1,331 @@
+import numpy as np
+import pandas as pd
+import logging
+from typing import List, Optional
+from pathlib import Path
+
+from ..config.ml_config import MLConfig
+from sklearn.preprocessing import PolynomialFeatures
+
+logger = logging.getLogger(__name__)
+
+
+class FeatureEngineering:
+    """Class for advanced feature engineering with feature selection."""
+
+    def __init__(self, config: MLConfig):
+        self.config = config
+        self.feature_store = {}
+        self.selected_features = None
+        self.feature_selector = None
+
+    def create_temporal_features(self, df: pd.DataFrame):
+        """Create temporal features from index."""
+        df_temp = df.copy()
+
+        try:
+            # Try to convert index to DatetimeIndex
+            index = pd.to_datetime(df_temp.index)
+
+            df_temp['year'] = index.year
+            df_temp['month'] = index.month
+            df_temp['day'] = index.day
+            df_temp['dayofweek'] = index.dayofweek
+            df_temp['dayofyear'] = index.dayofyear
+            df_temp['week'] = index.isocalendar().week.astype(int)
+            df_temp['quarter'] = index.quarter
+            df_temp['is_month_start'] = index.is_month_start.astype(int)
+            df_temp['is_month_end'] = index.is_month_end.astype(int)
+            df_temp['is_quarter_start'] = index.is_quarter_start.astype(int)
+            df_temp['is_quarter_end'] = index.is_quarter_end.astype(int)
+            df_temp['is_year_start'] = index.is_year_start.astype(int)
+            df_temp['is_year_end'] = index.is_year_end.astype(int)
+            df_temp['is_weekend'] = (index.dayofweek >= 5).astype(int)
+
+            logger.info("Created 15 temporal features from index")
+            return df_temp
+
+        except Exception as e:
+            logger.warning(f"Failed to create temporal features: {e}")
+            return df
+
+    def create_lag_features(self, df: pd.DataFrame, target_column: str,
+                           lags: List[int] = None, group_by: str = None):
+        """Create lag features."""
+        if lags is None:
+            lags = [1, 2, 3, 7, 14, 30]
+
+        df_lag = df.copy()
+
+        try:
+            if group_by and group_by in df_lag.columns:
+                for lag in lags:
+                    df_lag[f'{target_column}_lag_{lag}'] = df_lag.groupby(group_by)[target_column].shift(lag)
+            else:
+                for lag in lags:
+                    df_lag[f'{target_column}_lag_{lag}'] = df_lag[target_column].shift(lag)
+
+            logger.info(f"Created {len(lags)} lag features for {target_column}")
+
+        except Exception as e:
+            logger.warning(f"Error creating lag features: {e}")
+
+        return df_lag
+
+    def create_rolling_features(self, df: pd.DataFrame, target_column: str,
+                               windows: List[int] = None, group_by: str = None):
+        """Create rolling statistics."""
+        if windows is None:
+            windows = [3, 7, 14, 30]
+
+        df_rolling = df.copy()
+
+        try:
+            if group_by and group_by in df_rolling.columns:
+                for window in windows:
+                    df_rolling[f'{target_column}_rolling_mean_{window}'] = (
+                        df_rolling.groupby(group_by)[target_column]
+                        .rolling(window=window, min_periods=1)
+                        .mean()
+                        .reset_index(level=0, drop=True)
+                    )
+                    df_rolling[f'{target_column}_rolling_std_{window}'] = (
+                        df_rolling.groupby(group_by)[target_column]
+                        .rolling(window=window, min_periods=1)
+                        .std()
+                        .reset_index(level=0, drop=True)
+                    )
+                    df_rolling[f'{target_column}_rolling_min_{window}'] = (
+                        df_rolling.groupby(group_by)[target_column]
+                        .rolling(window=window, min_periods=1)
+                        .min()
+                        .reset_index(level=0, drop=True)
+                    )
+                    df_rolling[f'{target_column}_rolling_max_{window}'] = (
+                        df_rolling.groupby(group_by)[target_column]
+                        .rolling(window=window, min_periods=1)
+                        .max()
+                        .reset_index(level=0, drop=True)
+                    )
+            else:
+                for window in windows:
+                    df_rolling[f'{target_column}_rolling_mean_{window}'] = (
+                        df_rolling[target_column].rolling(window=window, min_periods=1).mean()
+                    )
+                    df_rolling[f'{target_column}_rolling_std_{window}'] = (
+                        df_rolling[target_column].rolling(window=window, min_periods=1).std()
+                    )
+                    df_rolling[f'{target_column}_rolling_min_{window}'] = (
+                        df_rolling[target_column].rolling(window=window, min_periods=1).min()
+                    )
+                    df_rolling[f'{target_column}_rolling_max_{window}'] = (
+                        df_rolling[target_column].rolling(window=window, min_periods=1).max()
+                    )
+
+            logger.info(f"Created {len(windows) * 4} rolling features for {target_column}")
+
+        except Exception as e:
+            logger.warning(f"Error creating rolling features: {e}")
+
+        return df_rolling
+
+    def create_interaction_features(self, df: pd.DataFrame,
+                                   numeric_columns: List[str] = None):
+        """Create interaction features with limits."""
+        if numeric_columns is None:
+            numeric_columns = df.select_dtypes(include=[np.number]).columns.tolist()
+
+        if len(numeric_columns) < 2:
+            return df
+
+        df_interaction = df.copy()
+        interactions_created = 0
+        max_interactions = min(100, len(numeric_columns) * (len(numeric_columns) - 1) // 2)
+
+        try:
+            # Limit number of created interaction features
+            for i in range(min(len(numeric_columns), 20)):
+                for j in range(i + 1, min(len(numeric_columns), 20)):
+                    if interactions_created >= max_interactions:
+                        break
+
+                    col1, col2 = numeric_columns[i], numeric_columns[j]
+
+                    # Create only if features are not too highly correlated
+                    corr = abs(df_interaction[col1].corr(df_interaction[col2]))
+                    if corr < 0.95:  # Don't create for highly correlated features
+                        df_interaction[f'{col1}_x_{col2}'] = df_interaction[col1] * df_interaction[col2]
+                        interactions_created += 1
+
+            logger.info(f"Created {interactions_created} interaction features")
+
+        except Exception as e:
+            logger.warning(f"Error creating interaction features: {e}")
+
+        return df_interaction
+
+    def create_polynomial_features(self, df: pd.DataFrame,
+                                  numeric_columns: List[str] = None,
+                                  degree: int = 2):
+        """Create polynomial features."""
+        if not self.config.create_polynomial_features:
+            return df
+
+        if numeric_columns is None:
+            numeric_columns = df.select_dtypes(include=[np.number]).columns.tolist()
+
+        df_poly = df.copy()
+
+        try:
+            # Limit number of features for polynomial transformation
+            if len(numeric_columns) > 10:
+                numeric_columns = numeric_columns[:10]
+                logger.info(f"Limited number of features for polynomial transformation to 10")
+
+            poly = PolynomialFeatures(
+                degree=min(degree, 3),  # Limit degree
+                include_bias=False,
+                interaction_only=False
+            )
+
+            poly_features = poly.fit_transform(df_poly[numeric_columns])
+            poly_feature_names = poly.get_feature_names_out(numeric_columns)
+
+            poly_df = pd.DataFrame(
+                poly_features,
+                columns=poly_feature_names,
+                index=df_poly.index
+            )
+
+            df_poly = pd.concat([df_poly, poly_df], axis=1)
+
+            logger.info(f"Created {len(poly_feature_names)} polynomial features of degree {degree}")
+
+        except Exception as e:
+            logger.warning(f"Error creating polynomial features: {e}")
+
+        return df_poly
+
+    def select_features(self, X: pd.DataFrame, y: pd.Series, method: str = None):
+        """Select most important features."""
+        if not self.config.use_feature_selection:
+            self.selected_features = list(X.columns)
+            return X
+
+        if method is None:
+            method = self.config.feature_selection_method
+
+        try:
+            if method == 'importance':
+                # Use random forest for feature importance assessment
+                from sklearn.ensemble import RandomForestRegressor
+                rf = RandomForestRegressor(n_estimators=100, random_state=self.config.random_state, n_jobs=-1)
+                rf.fit(X, y)
+                importances = rf.feature_importances_
+
+                # Determine number of features to select
+                if isinstance(self.config.n_features_to_select, float):
+                    n_features = int(X.shape[1] * self.config.n_features_to_select)
+                else:
+                    n_features = min(self.config.n_features_to_select, X.shape[1])
+
+                # Select top-n features
+                indices = np.argsort(importances)[-n_features:]
+                self.selected_features = list(X.columns[indices])
+                self.feature_selector = lambda X_sel: X_sel.iloc[:, indices]
+
+                logger.info(f"Selected {len(self.selected_features)} features using importance method")
+
+            elif method == 'correlation':
+                # Select based on correlation with target
+                correlations = X.apply(lambda col: col.corr(y))
+
+                if isinstance(self.config.n_features_to_select, float):
+                    n_features = int(X.shape[1] * self.config.n_features_to_select)
+                else:
+                    n_features = min(self.config.n_features_to_select, X.shape[1])
+
+                # Select features with highest absolute correlation
+                indices = np.argsort(np.abs(correlations))[-n_features:]
+                self.selected_features = list(X.columns[indices])
+                self.feature_selector = lambda X_sel: X_sel.iloc[:, indices]
+
+                logger.info(f"Selected {len(self.selected_features)} features using correlation method")
+
+            elif method == 'variance':
+                # Select based on variance
+                from sklearn.feature_selection import VarianceThreshold
+                selector = VarianceThreshold(threshold=0.01)
+                selector.fit(X)
+                self.selected_features = list(X.columns[selector.get_support()])
+                self.feature_selector = selector
+
+                logger.info(f"Selected {len(self.selected_features)} features using variance method")
+
+            # Apply selection
+            if self.selected_features:
+                X_selected = X[self.selected_features]
+                return X_selected
+            else:
+                return X
+
+        except Exception as e:
+            logger.warning(f"Error selecting features: {e}")
+            return X
+
+    def apply_feature_engineering(self, df: pd.DataFrame, target_column: str):
+        """Apply all feature engineering methods with constraints."""
+        if not self.config.feature_engineering:
+            return df
+
+        logger.info("Applying feature engineering...")
+
+        df_engineered = df.copy()
+
+        try:
+            # Save original number of features
+            original_features = df_engineered.shape[1]
+
+            # Create temporal features from index
+            df_engineered = self.create_temporal_features(df_engineered)
+
+            # Limit polynomial feature creation
+            if self.config.create_polynomial_features:
+                df_engineered = self.create_polynomial_features(
+                    df_engineered,
+                    degree=self.config.polynomial_degree
+                )
+
+            # Limit interaction feature creation
+            if self.config.create_interaction_features:
+                df_engineered = self.create_interaction_features(df_engineered)
+
+            # Remove duplicate columns
+            df_engineered = df_engineered.loc[:, ~df_engineered.columns.duplicated()]
+
+            # Remove features with constant values
+            constant_cols = [col for col in df_engineered.columns if df_engineered[col].nunique() <= 1]
+            if constant_cols:
+                df_engineered = df_engineered.drop(columns=constant_cols)
+                logger.info(f"Removed {len(constant_cols)} features with constant values")
+
+            logger.info(f"After feature engineering: {df_engineered.shape[1]} features (was {original_features})")
+
+            return df_engineered
+
+        except Exception as e:
+            logger.error(f"Feature engineering error: {e}")
+            return df
+
+    def save_features(self, features: pd.DataFrame, name: str):
+        """Save features to feature store."""
+        path = f"{self.config.feature_store_dir}/{name}.parquet"
+        features.to_parquet(path)
+        self.feature_store[name] = path
+        logger.info(f"Features saved to feature store: {path}")
+
+    def load_features(self, name: str):
+        """Load features from feature store."""
+        if name in self.feature_store:
+            return pd.read_parquet(self.feature_store[name])
+        return None

src/ml_pipeline/features/selection.py

@@ -0,0 +1,86 @@
+import numpy as np
+import pandas as pd
+import logging
+from typing import List, Optional
+from sklearn.feature_selection import VarianceThreshold
+
+logger = logging.getLogger(__name__)
+
+
+class FeatureSelector:
+    """Feature selection using various methods."""
+    
+    def __init__(self, config):
+        self.config = config
+        self.selected_features = None
+        self.feature_selector = None
+    
+    def select_features(self, X: pd.DataFrame, y: pd.Series, method: str = None) -> pd.DataFrame:
+        """Select the most relevant features using the specified method."""
+        if not self.config.use_feature_selection:
+            self.selected_features = list(X.columns)
+            return X
+
+        if method is None:
+            method = self.config.feature_selection_method
+
+        try:
+            if method == 'importance':
+                # Use random forest for feature importance estimation
+                from sklearn.ensemble import RandomForestRegressor
+                rf = RandomForestRegressor(
+                    n_estimators=100, 
+                    random_state=self.config.random_state, 
+                    n_jobs=-1
+                )
+                rf.fit(X, y)
+                importances = rf.feature_importances_
+
+                # Determine number of features to select
+                if isinstance(self.config.n_features_to_select, float):
+                    n_features = int(X.shape[1] * self.config.n_features_to_select)
+                else:
+                    n_features = min(self.config.n_features_to_select, X.shape[1])
+
+                # Select top-n features
+                indices = np.argsort(importances)[-n_features:]
+                self.selected_features = list(X.columns[indices])
+                self.feature_selector = lambda X_sel: X_sel.iloc[:, indices]
+
+                logger.info(f"Selected {len(self.selected_features)} features using importance method")
+
+            elif method == 'correlation':
+                # Selection based on correlation with target variable
+                correlations = X.apply(lambda col: col.corr(y))
+
+                if isinstance(self.config.n_features_to_select, float):
+                    n_features = int(X.shape[1] * self.config.n_features_to_select)
+                else:
+                    n_features = min(self.config.n_features_to_select, X.shape[1])
+
+                # Select features with highest absolute correlation
+                indices = np.argsort(np.abs(correlations))[-n_features:]
+                self.selected_features = list(X.columns[indices])
+                self.feature_selector = lambda X_sel: X_sel.iloc[:, indices]
+
+                logger.info(f"Selected {len(self.selected_features)} features using correlation method")
+
+            elif method == 'variance':
+                # Selection based on feature variance
+                selector = VarianceThreshold(threshold=0.01)
+                selector.fit(X)
+                self.selected_features = list(X.columns[selector.get_support()])
+                self.feature_selector = selector
+
+                logger.info(f"Selected {len(self.selected_features)} features using variance method")
+
+            # Apply selection
+            if self.selected_features:
+                X_selected = X[self.selected_features]
+                return X_selected
+            else:
+                return X
+
+        except Exception as e:
+            logger.warning(f"Feature selection error: {e}")
+            return X

src/ml_pipeline/models/boosting_models.py

@@ -0,0 +1,514 @@
+import numpy as np
+import pandas as pd
+import re
+import logging
+import inspect
+from datetime import datetime
+import traceback
+
+from src.ml_pipeline.config.ml_config import MLConfig
+from src.ml_pipeline.utils.helpers import _filter_fit_kwargs, _filter_kwargs_for_callable
+
+from ..core.base_model import BaseMLModel
+
+# Import boosting libraries
+try:
+    import xgboost as xgb
+    XGBOOST_AVAILABLE = True
+except ImportError:
+    XGBOOST_AVAILABLE = False
+
+try:
+    import lightgbm as lgb
+    LIGHTGBM_AVAILABLE = True
+except ImportError:
+    LIGHTGBM_AVAILABLE = False
+
+try:
+    from catboost import CatBoostRegressor
+    CATBOOST_AVAILABLE = True
+except ImportError:
+    CATBOOST_AVAILABLE = False
+
+from sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor
+
+logger = logging.getLogger(__name__)
+
+
+# Gradient Boosting
+class GradientBoostingModel(BaseMLModel):
+    def __init__(self, config: MLConfig):
+        super().__init__(config, 'gradient_boosting')
+
+    def create_model(self, **kwargs):
+        params = {
+            'loss': 'squared_error',
+            'learning_rate': 0.1,
+            'n_estimators': 100,
+            'subsample': 1.0,
+            'criterion': 'friedman_mse',
+            'min_samples_split': 2,
+            'min_samples_leaf': 1,
+            'max_depth': 3,
+            'min_impurity_decrease': 0.0,
+            'init': None,
+            'random_state': self.config.random_state,
+            'max_features': None,
+            'alpha': 0.9,
+            'verbose': 0,
+            'max_leaf_nodes': None,
+            'warm_start': False,
+            'validation_fraction': 0.1,
+            'n_iter_no_change': None,
+            'tol': 0.0001,
+            'ccp_alpha': 0.0
+        }
+        params.update(kwargs)
+        self.model = GradientBoostingRegressor(**params)
+        return self.model
+
+
+class HistGradientBoostingModel(BaseMLModel):
+    def __init__(self, config: MLConfig):
+        super().__init__(config, 'hist_gradient_boosting')
+
+    def create_model(self, **kwargs):
+        params = {
+            'loss': 'squared_error',
+            'learning_rate': 0.1,
+            'max_iter': 100,
+            'max_leaf_nodes': 31,
+            'max_depth': None,
+            'min_samples_leaf': 20,
+            'l2_regularization': 0.0,
+            'max_bins': 255,
+            'categorical_features': None,
+            'monotonic_cst': None,
+            'warm_start': False,
+            'early_stopping': 'auto',
+            'scoring': 'loss',
+            'validation_fraction': 0.1,
+            'n_iter_no_change': 10,
+            'tol': 1e-07,
+            'verbose': 0,
+            'random_state': self.config.random_state
+        }
+        params.update(kwargs)
+        self.model = HistGradientBoostingRegressor(**params)
+        return self.model
+
+
+# LightGBM Model
+class LightGBMModel(BaseMLModel):
+    def __init__(self, config: MLConfig):
+        super().__init__(config, 'lightgbm')
+        self.feature_names_ = None  # Store feature names for LGBM
+        self._original_to_clean_map = None
+
+    def clean_feature_names_for_lgbm(self, X):
+        """
+        Clean feature names for LightGBM.
+        Replace prohibited characters with '_' and make names unique.
+        Logs changes and saves original->clean mapping.
+        """
+        if X is None:
+            return None
+
+        if not hasattr(X, 'columns'):
+            return X  # non-DataFrame - do nothing
+
+        X_clean = X.copy()
+        orig_names = list(X_clean.columns)
+        clean_names = []
+        orig_to_clean = {}
+
+        for i, name in enumerate(orig_names):
+            s = str(name)
+            # Replace special characters with underscore
+            for ch in ['[', ']', '<', '>', '=', ' ', ',', ';', ':', '(', ')', '{', '}', '/', '\\', '\t', '\n']:
+                s = s.replace(ch, '_')
+            # Keep only alphanumeric, underscores, and hyphens
+            while '__' in s:
+                s = s.replace('__', '_')
+            s = s.strip('_')
+            if not s:
+                s = f'feature_{i}'
+            clean_names.append(s)
+            orig_to_clean[orig_names[i]] = s
+
+        # Make names unique (add suffixes for collisions)
+        seen = {}
+        final_names = []
+        for name in clean_names:
+            base = name
+            cnt = seen.get(base, 0)
+            if cnt:
+                name = f"{base}_{cnt}"
+            seen[base] = cnt + 1
+            final_names.append(name)
+
+        # Log changes if any
+        if orig_names != final_names:
+            logger.debug(
+                f"Feature names adapted for LightGBM compatibility. "
+                f"Example changes: {list(orig_to_clean.items())[:10]}"
+            )
+
+        X_clean.columns = final_names
+        self.feature_names_ = final_names
+        self._original_to_clean_map = orig_to_clean
+        return X_clean
+
+    def create_model(self, **kwargs):
+        """Create LightGBM model with only constructor-compatible kwargs."""
+        base_params = {
+            'boosting_type': 'gbdt',
+            'num_leaves': 31,
+            'max_depth': -1,
+            'learning_rate': 0.1,
+            'n_estimators': 100,
+            'objective': 'regression',
+            'random_state': self.config.random_state,
+            'importance_type': 'split',
+            'verbosity': -1,  # use verbosity instead of verbose
+            'n_jobs': getattr(self.config, "n_jobs", None)
+        }
+
+        # GPU options (if needed)
+        if getattr(self.config, 'use_gpu', False):
+            base_params.update({
+                'device': 'gpu',
+                'gpu_platform_id': 0,
+                'gpu_device_id': 0
+            })
+
+        # Filter kwargs, keeping only those accepted by __init__
+        init_filtered, removed_init = _filter_kwargs_for_callable(lgb.LGBMRegressor.__init__, kwargs, logger=logger, obj_name="LGBMRegressor.__init__")
+        if removed_init:
+            logger.debug(f"Parameters not suitable for LGBMRegressor constructor and omitted: {removed_init}")
+
+        # Merge constructor parameters, priority to init_filtered
+        for k, v in init_filtered.items():
+            base_params[k] = v
+
+        # Ensure 'verbose' is not passed to constructor or fit
+        if 'verbose' in base_params:
+            del base_params['verbose']
+
+        # Create model
+        self.model = lgb.LGBMRegressor(**{k: v for k, v in base_params.items() if v is not None})
+        return self.model
+
+    def fit(self, X_train, y_train, X_val=None, y_val=None, **kwargs):
+        """Train LightGBM model with proper constructor and fit parameter handling."""
+        if 'verbose' in kwargs:
+            logger.debug("Parameter 'verbose' removed from fit() for LightGBM")
+            kwargs.pop('verbose', None)
+
+        start_time = datetime.now()
+        try:
+            # Clean feature names
+            X_train_clean = self.clean_feature_names_for_lgbm(X_train)
+
+            # Prepare validation parameters
+            fit_specific = {}
+            if X_val is not None and y_val is not None:
+                X_val_clean = self.clean_feature_names_for_lgbm(X_val)
+                fit_specific['eval_set'] = [(X_val_clean, y_val)]
+                fit_specific['eval_names'] = ['valid']
+
+                eval_metric = kwargs.pop('eval_metric', 'l2')
+                if eval_metric == 'l2':
+                    fit_specific['eval_metric'] = 'l2'
+                elif eval_metric == 'mae':
+                    fit_specific['eval_metric'] = 'l1'
+                else:
+                    fit_specific['eval_metric'] = eval_metric
+
+            # Early stopping
+            early_stopping_rounds = kwargs.pop('early_stopping_rounds', None)
+            if early_stopping_rounds and X_val is not None:
+                fit_specific['early_stopping_rounds'] = early_stopping_rounds
+
+            # Callbacks
+            callbacks = kwargs.pop('callbacks', None)
+            if callbacks:
+                fit_specific['callbacks'] = callbacks
+
+            # Separate kwargs into constructor parameters and remaining (for fit)
+            init_kwargs, removed_init = _filter_kwargs_for_callable(lgb.LGBMRegressor.__init__, kwargs, logger=logger, obj_name="LGBMRegressor.__init__")
+            
+            # Create model (if not already created) with init_kwargs
+            if self.model is None:
+                self.create_model(**init_kwargs)
+            else:
+                # If model already exists, log that some init_kwargs were ignored
+                if init_kwargs:
+                    logger.debug(f"Constructor parameters passed after model creation: {list(init_kwargs.keys())}")
+
+            # Remaining kwargs (since init_kwargs were taken from kwargs) - potentially fit parameters
+            remaining_kwargs = {k: v for k, v in kwargs.items() if k not in init_kwargs}
+
+            # Combine parameters for fit and filter against model.fit signature
+            all_fit_kwargs = {**remaining_kwargs, **fit_specific}
+            filtered_fit_kwargs, removed_fit = _filter_kwargs_for_callable(self.model.fit, all_fit_kwargs, logger=logger, obj_name=f"{self.model.__class__.__name__}.fit")
+
+            # Ensure 'verbose' is removed - LightGBM expects 'verbosity' in constructor
+            if 'verbose' in filtered_fit_kwargs:
+                filtered_fit_kwargs.pop('verbose', None)
+            # Also remove constructor parameters that might have been passed by mistake
+            for p in ['boosting_type', 'num_leaves', 'learning_rate', 'n_estimators', 'objective', 'verbosity', 'n_jobs']:
+                if p in filtered_fit_kwargs:
+                    filtered_fit_kwargs.pop(p, None)
+
+            if removed_fit:
+                logger.debug(f"Parameters removed from fit call for {self.model.__class__.__name__}: {removed_fit}")
+
+            # Actual training call
+            self.model.fit(X_train_clean, y_train, **filtered_fit_kwargs)
+
+            # Save training information
+            self.training_time = (datetime.now() - start_time).total_seconds()
+            self.is_fitted = True
+
+            # Predictions on training data
+            self.predictions['train'] = self.predict(X_train)
+            self.residuals['train'] = y_train - self.predictions['train']
+
+            # Predictions on validation data
+            if X_val is not None and y_val is not None:
+                self.predictions['val'] = self.predict(X_val)
+                self.residuals['val'] = y_val - self.predictions['val']
+
+            logger.info(f"✓ LightGBM trained in {self.training_time:.2f} seconds")
+
+            # Save feature importance
+            if hasattr(self.model, 'feature_importances_'):
+                self.feature_importances_ = self.model.feature_importances_
+
+            return self
+
+        except Exception as e:
+            logger.error(f"✗ Error training LightGBM: {str(e)}")
+            if getattr(self.config, "debug", False):
+                logger.error(traceback.format_exc())
+            raise
+
+    def predict(self, X):
+        """Prediction with feature name cleaning."""
+        if not self.is_fitted:
+            raise ValueError("Model is not trained")
+
+        start_time = datetime.now()
+        X_clean = self.clean_feature_names_for_lgbm(X)
+
+        # If pipeline exists and expects original/cleaned names - use it
+        if hasattr(self, 'pipeline') and self.pipeline is not None:
+            preds = self.pipeline.predict(X_clean)
+        else:
+            preds = self.model.predict(X_clean)
+
+        # Save inference time
+        inference_time_ms = (datetime.now() - start_time).total_seconds() * 1000
+        try:
+            n = len(X)
+        except Exception:
+            n = 1
+        self.inference_time = inference_time_ms / n if n > 0 else inference_time_ms
+
+        return preds
+
+    def get_feature_importance(self, feature_names=None):
+        """Return feature importance."""
+        if not self.is_fitted:
+            raise ValueError("Model is not trained")
+
+        if hasattr(self.model, 'feature_importances_'):
+            importances = self.model.feature_importances_
+
+            if feature_names is None and self.feature_names_ is not None:
+                feature_names = self.feature_names_
+
+            if feature_names is not None and len(feature_names) == len(importances):
+                return dict(zip(feature_names, importances))
+            else:
+                return importances
+
+        return None
+
+
+class XGBoostModel(BaseMLModel):
+    def __init__(self, config: MLConfig):
+        super().__init__(config, 'xgboost')
+
+    def create_model(self, **params):
+        params.setdefault('n_estimators', 100)
+        params.setdefault('max_depth', 3)
+        params.setdefault('learning_rate', 0.1)
+        params.setdefault('verbosity', 0)
+        params.setdefault('objective', 'reg:squarederror')
+        params.setdefault('booster', 'gbtree')
+        params.setdefault('tree_method', 'auto')
+        params.setdefault('n_jobs', self.config.n_jobs)
+        params.setdefault('gamma', 0)
+        params.setdefault('min_child_weight', 1)
+        params.setdefault('subsample', 1)
+        params.setdefault('colsample_bytree', 1)
+        params.setdefault('reg_alpha', 0)
+        params.setdefault('reg_lambda', 1)
+        params.setdefault('scale_pos_weight', 1)
+        params.setdefault('random_state', self.config.random_state)
+        params.setdefault('missing', np.nan)
+
+        if self.config.use_gpu:
+            params['tree_method'] = 'gpu_hist'
+            params['predictor'] = 'gpu_predictor'
+
+        self.model = xgb.XGBRegressor(**params)
+        return self.model
+
+    def fit(self, X_train, y_train, X_val=None, y_val=None, **kwargs):
+        start_time = datetime.now()
+        try:
+            # Extract typical fit-only parameters (won't be passed to create_model)
+            early_stopping = kwargs.pop('early_stopping_rounds', None)
+            fit_verbose = kwargs.pop('fit_verbose', False)  # User setting for fit verbosity
+
+            # Clean features
+            X_train_clean = self.clean_feature_names(X_train)
+            X_val_clean = None
+            if X_val is not None:
+                X_val_clean = self.clean_feature_names(X_val)
+
+            # Create model (remaining kwargs are constructor parameters)
+            if self.model is None:
+                self.create_model(**kwargs)
+
+            # Collect arguments for fit
+            fit_args = {}
+            if X_val_clean is not None and y_val is not None:
+                fit_args['eval_set'] = [(X_val_clean, y_val)]
+            # Try to use standard 'verbose' if model supports it
+            fit_args['verbose'] = fit_verbose
+
+            # Handle early stopping - prefer direct parameter, otherwise use callback
+            if early_stopping and 'eval_set' in fit_args:
+                # If model.fit supports 'early_stopping_rounds' - pass directly
+                try_fit_args = fit_args.copy()
+                try_fit_args['early_stopping_rounds'] = early_stopping
+                filtered_try = _filter_fit_kwargs(self.model, try_fit_args)
+                if 'early_stopping_rounds' in filtered_try:
+                    fit_args = filtered_try
+                else:
+                    # Fallback: use callback
+                    callbacks = try_fit_args.get('callbacks', [])
+                    callbacks = list(callbacks)  # Copy
+                    try:
+                        callbacks.append(xgb.callback.EarlyStopping(rounds=early_stopping, save_best=True))
+                        fit_args['callbacks'] = callbacks
+                    except Exception:
+                        # If callback unavailable - log only (train without early stopping)
+                        logger.warning("Failed to add xgb.callback.EarlyStopping - early stopping not applied")
+            else:
+                # If early stopping not specified but in config and eval_set exists
+                cfg_es = getattr(self.config, 'early_stopping_rounds', None)
+                if cfg_es and 'eval_set' in fit_args:
+                    try_fit_args = fit_args.copy()
+                    try_fit_args['early_stopping_rounds'] = cfg_es
+                    filtered_try = _filter_fit_kwargs(self.model, try_fit_args)
+                    if 'early_stopping_rounds' in filtered_try:
+                        fit_args = filtered_try
+                    else:
+                        callbacks = try_fit_args.get('callbacks', [])
+                        callbacks = list(callbacks)
+                        try:
+                            callbacks.append(xgb.callback.EarlyStopping(rounds=cfg_es, save_best=True))
+                            fit_args['callbacks'] = callbacks
+                        except Exception:
+                            logger.debug("Failed to set early stopping callback for xgboost")
+
+            # Final filtering of fit_args against model.fit signature
+            fit_args = _filter_fit_kwargs(self.model, fit_args)
+
+            # Fit call
+            self.model.fit(X_train_clean, y_train, **fit_args)
+
+            self.training_time = (datetime.now() - start_time).total_seconds()
+            self.is_fitted = True
+
+            # Predictions on cleaned data
+            self.predictions['train'] = self.model.predict(X_train_clean)
+            self.residuals['train'] = y_train - self.predictions['train']
+
+            if X_val_clean is not None and y_val is not None:
+                self.predictions['val'] = self.model.predict(X_val_clean)
+                self.residuals['val'] = y_val - self.predictions['val']
+
+            logger.info(f"✓ XGBoost trained in {self.training_time:.2f} seconds")
+
+            if hasattr(self.model, 'best_iteration'):
+                self.best_params['best_iteration'] = self.model.best_iteration
+            if hasattr(self.model, 'best_score'):
+                self.best_params['best_score'] = self.model.best_score
+
+            return self
+
+        except Exception as e:
+            logger.error(f"✗ Error training XGBoost: {e}")
+            if self.config.debug:
+                logger.error(traceback.format_exc())
+            raise
+
+
+class CatBoostModel(BaseMLModel):
+    def __init__(self, config: MLConfig):
+        super().__init__(config, 'catboost')
+
+    def create_model(self, **kwargs):
+        params = {
+            'iterations': 100,
+            'learning_rate': 0.1,
+            'depth': 6,
+            'l2_leaf_reg': 3,
+            'loss_function': 'RMSE',
+            'verbose': False,
+            'random_seed': self.config.random_state,
+            'bagging_temperature': 1,
+            'border_count': 254,
+            'thread_count': self.config.n_jobs if self.config.n_jobs != -1 else -1,
+            'task_type': 'GPU' if self.config.use_gpu else 'CPU',
+            'early_stopping_rounds': self.config.early_stopping_rounds
+        }
+        params.update(kwargs)
+        self.model = CatBoostRegressor(**params)
+        return self.model
+
+    def fit(self, X_train, y_train, X_val=None, y_val=None, **kwargs):
+        start_time = datetime.now()
+
+        try:
+            X_train_clean = self.clean_feature_names(X_train)
+
+            if X_val is not None and y_val is not None:
+                X_val_clean = self.clean_feature_names(X_val)
+                kwargs['eval_set'] = (X_val_clean, y_val)
+
+            self.model.fit(X_train_clean, y_train, **kwargs)
+
+            self.training_time = (datetime.now() - start_time).total_seconds()
+            self.is_fitted = True
+
+            self.predictions['train'] = self.predict(X_train)
+            self.residuals['train'] = y_train - self.predictions['train']
+
+            if X_val is not None and y_val is not None:
+                self.predictions['val'] = self.predict(X_val)
+                self.residuals['val'] = y_val - self.predictions['val']
+
+            logger.info(f"✓ CatBoost trained in {self.training_time:.2f} seconds")
+
+            return self
+
+        except Exception as e:
+            logger.error(f"✗ Error training CatBoost: {e}")
+            raise