README.md

# XAUUSD ML Trading System 🪙📈

A comprehensive ML-based Gold (XAUUSD) trading system implementing 5 institutional-grade strategies with online learning, Walk-Forward Optimization (WFO) backtesting, and Probability of Best Fit analysis.

## ⚠️ Disclaimer

**This is an educational/research project. This is NOT financial advice. Trading Gold (XAUUSD) involves substantial risk of loss. Past performance (including backtests) is NOT indicative of future results. Never trade with money you can't afford to lose.**

---

## 🏗️ Architecture

```
xauusd_trader/
├── data_loader.py              # XAUUSD data loading (yfinance + synthetic)
├── main.py                     # Main orchestrator - ties everything together
├── strategies/
│   ├── smc_strategy.py         # Strategy 1: Smart Money Concepts (Order Blocks, BOS, FVG)
│   ├── ict_power_of_3.py       # Strategy 2: ICT Power of 3 (AMD phases)
│   ├── momentum_strategy.py    # Strategy 3: RSI Divergence & Stochastics
│   ├── sentiment_strategy.py   # Strategy 4: Fundamental/Sentiment Alignment
│   └── risk_management.py      # Strategy 5: Volatility-aware Risk Management
├── ml/
│   └── ensemble_model.py       # XGBoost ensemble with online learning & drift detection
└── backtesting/
    ├── wfo_engine.py           # Walk-Forward Optimization backtesting
    └── probability_best_fit.py # Distribution fitting & strategy selection
```

## 📊 5 Trading Strategies

### 1. Smart Money Concepts (SMC) - Order Block Tap Entry
The most profitable strategy in our analysis. Detects institutional footprints:
- **Break of Structure (BOS)**: Identifies when price breaks key swing highs/lows
- **Order Blocks**: Last opposing candle before a displacement that broke structure
- **Liquidity Sweeps**: Wick-through of swing points (stop hunts)
- **Fair Value Gaps (FVG)**: Imbalance zones in 3-candle patterns
- **50% Equilibrium Entry**: Limit orders at OB midpoint with tight stops behind the wick

### 2. ICT Power of 3 (AMD)
Session-based institutional movement detection:
- **Accumulation (Asian session)**: Tight consolidation range detection
- **Manipulation (London open)**: False breakout / liquidity sweep identification  
- **Distribution (NY session)**: True directional move after manipulation
- Entry only during Distribution phase after Manipulation confirmation

### 3. RSI Divergence & Stochastics
Momentum oscillator confluence:
- **Standard RSI Divergence**: Price/RSI divergence for reversal signals
- **Hidden RSI Divergence**: Trend continuation signals
- **Stochastic Crossovers**: Precise timing in trending markets
- **Moving Average Alignment**: EMA 9/21/50/100/200 crossover confirmation
- **Bollinger Bands + MACD**: Additional confluence factors

### 4. Fundamental/Sentiment Alignment
Macro-driven directional bias:
- **Risk-On/Risk-Off Score**: Composite of VIX, DXY, SPY, TLT
- **Dollar Index (DXY)** inverse correlation
- **VIX Fear Gauge** positive correlation during crises
- **Bond Yields** inverse correlation proxy
- Falls back to price-action derived sentiment when macro data unavailable

### 5. Strict Risk Management
Gold-specific volatility protection:
- **ATR-based Dynamic Stops**: Min/max stop distance in ATR multiples
- **Volatility-adjusted Position Sizing**: Smaller positions in high-vol regimes
- **Drawdown-based Risk Scaling**: Progressive reduction at 1%/2%/3%/5% DD levels
- **Session-aware Risk**: Lower exposure during Asian session and off-hours
- **Daily Loss Limit**: 3% daily cap, 10% total DD halt
- **Guaranteed Stop Logic**: Buffer for gap protection

## 🤖 ML Engine

### XGBoost Ensemble with Online Learning
- **3-model ensemble** with diverse random seeds and subsampling
- **86 features** across all 5 strategy groups
- **Hedge algorithm** for dynamic model weighting based on recent performance
- **ADWIN drift detection** for market regime changes
- **Incremental warm-start retraining** after every N closed trades
- **RobustScaler** preprocessing for outlier resilience

### Feature Groups (86 total)
| Group | Features | Key Indicators |
|-------|----------|---------------|
| SMC | 16 | BOS, OB strength, FVG, liquidity sweeps, structure code |
| AMD | 14 | Accumulation tightness, manipulation direction/strength, distribution |
| Momentum | 31 | RSI (7/14/21), Stochastics, MACD, BBands, EMA crossovers, divergences |
| Sentiment | 5 | Risk sentiment score, trend, regime code |
| Volatility | 11 | HV, Parkinson, Yang-Zhang, vol ratio, gap detection |
| Price Action | 8 | Returns, candle body/wicks, ATR |
| Session | 1 | Trading session code |

## 📈 Walk-Forward Optimization (WFO)

Rolling temporal cross-validation that prevents look-ahead bias:

```
Fold 0: [====TRAIN====][=TEST=]
Fold 1: [======TRAIN======][=TEST=]
Fold 2: [=========TRAIN=========][=TEST=]
```

### Key Metrics Tracked
- **WFO Efficiency** = OOS Sharpe / IS Sharpe (measures overfitting)
- **Sharpe Stability** = Mean(OOS Sharpe) / Std(OOS Sharpe)
- **% Profitable Folds**: Consistency across market conditions
- Per-fold: Win Rate, Profit Factor, Max Drawdown, R-multiple

## 🎯 Probability of Best Fit

Statistical strategy selection using:

1. **Distribution Fitting**: Fits Normal, Skew-Normal, Student-t, Laplace, Logistic to each strategy's P&L series. Ranked by BIC.
2. **Monte Carlo Simulation**: 10,000 resampled forward paths → probability of positive P&L
3. **OOS Consistency**: % of WFO folds where strategy was profitable
4. **Sharpe Stability**: Risk-adjusted return consistency
5. **Hedge Ensemble Weights**: Distribution-free exponential weighting that provably converges to best strategy

### Composite Score
```
Score = 0.25 × Normalized_Expected_Return 
      + 0.25 × Pct_Positive_Folds
      + 0.25 × Sharpe_Stability
      + 0.25 × MC_Probability
```

## 🚀 Quick Start

### Installation
```bash
pip install numpy pandas scikit-learn xgboost scipy yfinance river joblib
```

### Run Full Pipeline
```python
from xauusd_trader.main import XAUUSDTradingSystem, ModelConfig, BacktestConfig
from xauusd_trader.strategies.risk_management import RiskConfig

system = XAUUSDTradingSystem(
    model_config=ModelConfig(
        n_estimators=300,
        max_depth=6,
        learning_rate=0.08,
        n_models=3,
        online_batch_size=30,
    ),
    risk_config=RiskConfig(
        initial_capital=100000.0,
        max_risk_per_trade=0.01,       # 1% risk per trade
        max_daily_loss_pct=0.03,        # 3% daily loss limit
        max_total_drawdown_pct=0.10,    # 10% total DD halt
    ),
    backtest_config=BacktestConfig(
        n_folds=5,
        forward_bars=10,
        min_confidence=0.35,
    ),
)

# Run everything: load data → features → train → WFO → Best Fit → Online trading
results = system.full_pipeline(use_real_data=True)
```

### Use Individual Components
```python
# Just run SMC analysis
from xauusd_trader.data_loader import load_xauusd, add_atr
from xauusd_trader.strategies.smc_strategy import compute_smc_features

df = load_xauusd(period="1y", interval="1h")
df = add_atr(df)
df = compute_smc_features(df)

# See Order Block signals
signals = df[df['smc_signal'] != 0]
print(signals[['close', 'smc_signal', 'smc_entry', 'smc_sl', 'smc_tp', 'smc_confidence']])
```

## 📊 Sample Backtest Results

Results from 5000-bar test (your results will vary with real data):

```
WFO BACKTEST (3 folds, all profitable):
  Total OOS Trades:  1260
  Win Rate:          52.2%
  Profit Factor:     2.09
  Sharpe Ratio:      5.34
  WFO Efficiency:    0.54
  Sharpe Stability:  3.07

STRATEGY RANKING (Probability of Best Fit):
  1. SMC_OrderBlock  Score=0.823  WinRate=81.8%  Sharpe=18.2  BestFit=Laplace
  2. ML_Ensemble     Score=0.782  WinRate=52.2%  Sharpe=5.36  BestFit=Normal
  3. Momentum        Score=0.490  WinRate=27.1%  Sharpe=0.37  BestFit=Student-t

ONLINE TRADING (with learning):
  Trades: 1621, WinRate: 51.3%, PF: 2.03, Sharpe: 4.98
  Drift events detected: 2, Model adapted successfully
```

## 🔧 Configuration

### Model Hyperparameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `n_estimators` | 300 | XGBoost trees per model |
| `max_depth` | 6 | Max tree depth |
| `learning_rate` | 0.08 | Boosting learning rate |
| `n_models` | 3 | Ensemble size |
| `online_batch_size` | 50 | Trades before incremental retrain |
| `online_boost_rounds` | 10 | Boost rounds per online update |

### Risk Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `max_risk_per_trade` | 1% | Max capital risk per trade |
| `max_daily_loss_pct` | 3% | Daily loss limit |
| `max_total_drawdown_pct` | 10% | Total DD halt threshold |
| `min_sl_atr_mult` | 0.5× ATR | Minimum stop loss distance |
| `max_sl_atr_mult` | 3.0× ATR | Maximum stop loss distance |

## 📚 References

- Smart Money Concepts (SMC) / Inner Circle Trader (ICT) methodology
- [Neural Network-Based Algorithmic Trading Systems](https://arxiv.org/abs/2508.02356)
- [Orchestration Framework for Financial Agents](https://arxiv.org/abs/2512.02227) - XGBoost + Rolling WFO
- [Ensembling Portfolio Strategies](https://arxiv.org/abs/2406.03652) - Universal Covers / Hedge algorithm
- [River: ML for Streaming Data](https://arxiv.org/abs/2012.04740) - Online learning framework
- [QTMRL](https://arxiv.org/abs/2508.20467) - RL for trading with technical indicators

## License

MIT License - See LICENSE file for details.