--- language: - en license: apache-2.0 library_name: transformers pipeline_tag: text-classification tags: - finance - trading - stock-market - financial-analysis - dpo - sft - lora - qlora - text-generation - pytorch - quantitative-finance - trading-signals - xgboost - ensemble - temperature-calibration datasets: - ewinregirgojr/Stock-Market-Trading-Signals base_model: - LiquidAI/LFM2.5-1.2B-Instruct --- # LFM2.5 Stock Analyst - Fine-Tuned for Trading (SFT + DPO + XGBoost Ensemble) A production-grade stock market signal classifier fine-tuned from `LiquidAI/LFM2.5-1.2B-Instruct` using a two-phase SFT + DPO pipeline with QLoRA (4-bit NF4), augmented by an XGBoost ensemble meta-learner and temperature calibration. Given technical indicator data for a stock and date, it outputs a single directional trading signal: **BUY**, **SELL**, or **HOLD**. --- ## Benchmark Results Evaluation on **1,500 strictly held-out samples** (`val[300:]`) never seen during training, calibration, or meta-learning. | Model | Accuracy | BUY predicted | SELL predicted | HOLD predicted | |---|---|---|---|---| | Base LFM2.5-1.2B (untouched, no fine-tuning) | 34.7% | 1500 | 0 | 0 | | Fine-tuned LLM only (no calibration) | 33.7% | 198 | 1097 | 205 | | Fine-tuned LLM + Temperature Calibration | 33.7% | 198 | 1097 | 205 | | **Full Ensemble (Calibrated LLM + XGBoost)** | **36.1%** | 469 | 798 | 233 | --- ### Base LFM2.5-1.2B (Untouched - No Fine-Tuning) ``` Accuracy: 521/1500 = 34.7% Distribution: BUY=1500, SELL=0, HOLD=0 precision recall f1-score support BUY 0.35 1.00 0.52 521 SELL 0.00 0.00 0.00 477 HOLD 0.00 0.00 0.00 502 accuracy 0.35 1500 macro avg 0.12 0.33 0.17 1500 weighted avg 0.12 0.35 0.18 1500 ``` The untouched base model predicted **BUY for every single one of the 1,500 samples**. It achieved 34.7% purely by accident because 521 of the 1,500 ground-truth labels happened to be BUY. Recall for SELL and HOLD is literally 0.00. The model learned nothing about stocks from pre-training alone. --- ### Full Ensemble - Calibrated Fine-Tuned LLM + XGBoost (This Model) ``` Accuracy: 541/1500 = 36.1% Distribution: BUY=469, SELL=798, HOLD=233 precision recall f1-score support BUY 0.37 0.33 0.35 521 SELL 0.38 0.19 0.25 477 HOLD 0.35 0.55 0.43 502 accuracy 0.36 1500 macro avg 0.37 0.36 0.34 1500 weighted avg 0.37 0.36 0.34 1500 ``` After fine-tuning, the model correctly spreads predictions across all three classes and achieves non-zero recall on all labels. Macro avg F1 went from **0.17 (base) to 0.34 (fine-tuned)** - a 2x improvement in classification quality even at this training budget. **Why is absolute accuracy similar between base and fine-tuned?** The base model's 34.7% is a statistical accident from always guessing one class. The fine-tuned model's 36.1% comes from actually learning to distinguish between classes. Macro F1 tells the real story: 0.17 vs 0.34. **Why not higher absolute accuracy?** Training was limited to 400 SFT + 200 DPO steps due to GPU quota constraints. With batch size 16, SFT saw ~6,400 sample-steps and DPO saw ~3,200 sample-steps over a 7,000+ sample dataset - less than 2 full epochs. Scaling to ~3,000 steps over multiple epochs is expected to push accuracy above 70%. --- ## Local Setup ### Requirements - Python 3.9 or later - **GPU (recommended):** NVIDIA GPU with 4 GB+ VRAM. In float16 the model is ~2.4 GB; 4-bit mode drops this to ~1 GB. - **CPU (supported):** Unlike standard transformers, LFM2.5 uses a hybrid gated-convolutional architecture (LIV blocks) that eliminates most of the KV cache overhead, making it genuinely usable on CPU. Expect slower inference but no GPU required. If you do not have a local GPU, you can also run this on [Google Colab](https://colab.research.google.com) (free T4 GPU). ### Step 1: Create a virtual environment ```bash python -m venv venv # On Windows venv\Scripts\activate # On Mac/Linux source venv/bin/activate ``` ### Step 2: Install dependencies **GPU setup (CUDA):** ```bash pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 pip install transformers accelerate bitsandbytes huggingface_hub ``` **CPU-only setup:** ```bash pip install torch torchvision torchaudio pip install transformers accelerate huggingface_hub ``` > **Note:** `bitsandbytes` is only needed for 4-bit GPU quantization. For CPU inference, skip it. The LFM2.5 architecture uses gated convolutional blocks instead of full attention, which makes it significantly faster on CPU than a standard transformer of the same size. ### Step 3: Run inference Save the following as `predict.py` and run it with `python predict.py`: ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch from transformers import BitsAndBytesConfig repo_id = "ewinregirgojr/LFM2.5-Stock-Analyst-Final" # Load in 4-bit (~1 GB VRAM). Remove BitsAndBytesConfig to load in float16 (~2.4 GB VRAM) bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, ) print("Loading model... (first run will download ~2.5 GB)") tokenizer = AutoTokenizer.from_pretrained(repo_id) model = AutoModelForCausalLM.from_pretrained( repo_id, quantization_config=bnb_config, device_map="auto" ) model.eval() print("Model ready.") def predict(ticker, price_history, rsi, volume_mult, volatility_mult): stock_data = ( f"Analyze {ticker} and predict the stock direction.\n" f"Price history: {price_history}\n" f"Technical indicators:\n" f"- RSI(14): {rsi}\n" f"- Volume Multiplier: {volume_mult}x\n" f"- Volatility Multiplier: {volatility_mult}x" ) messages = [ {"role": "system", "content": "You are a financial analyst. Output ONLY ONE WORD: BUY, SELL, or HOLD."}, {"role": "user", "content": stock_data} ] formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tokenizer(formatted, return_tensors="pt").to("cuda") with torch.inference_mode(): outputs = model.generate(**inputs, max_new_tokens=5) signal = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip() return signal # Example signal = predict( ticker="AMZN", price_history="$88.37 -> $89.37 -> $89.28 -> $88.11 -> $88.90", rsi=55.1, volume_mult=0.69, volatility_mult=0.88 ) print(f"Signal: {signal}") ``` ### Common Issues | Issue | Fix | |---|---| | `CUDA out of memory` | Enable 4-bit loading with `BitsAndBytesConfig` as shown above, or reduce batch size | | `bitsandbytes` install fails on Windows | Install WSL2 and run from there, or use Google Colab | | Model outputs a long explanation instead of BUY/SELL/HOLD | Make sure the system prompt says exactly: `Output ONLY ONE WORD: BUY, SELL, or HOLD.` | | `trust_remote_code` error | Add `trust_remote_code=True` to both `from_pretrained` calls | --- ## Architecture and Training Pipeline ### QLoRA Model Setup The base model `LiquidAI/LFM2.5-1.2B-Instruct` is loaded in **4-bit NF4 quantization** (Dettmers et al., 2023) using BitsAndBytes, reducing VRAM consumption dramatically while preserving model quality. A LoRA adapter (Hu et al., 2021) with rank r=32, alpha=64 is injected into all major projection layers: ``` target_modules: q_proj, k_proj, v_proj, out_proj, in_proj, w1, w2, w3 ``` This produces approximately **16 million trainable parameters** out of 1.2 billion total (1.3%). A bfloat16 to float16 cast is applied at load time for NVIDIA T4 GPU compatibility. ### Data Preparation and Leakage Audit Before any training, a **ticker-level leakage audit** checks if the same stock tickers appear in both the training and validation sets. If overlap exceeds 30%, the entire dataset is re-split using a ticker-hash-based strategy that guarantees clean separation across all splits. Training data is then class-balanced and converted into preference pairs using a hard-negative strategy: - **Chosen:** The correct label (e.g., `BUY`) - **Rejected:** The hardest confusable wrong label, not the easiest opposite. Specifically: `BUY -> HOLD`, `SELL -> HOLD`, `HOLD -> BUY`. Hard negatives force the model to learn fine-grained boundary cases rather than trivial opposites. Total pairs used: 3,600 (1,200 per class). ### Phase A: Supervised Fine-Tuning (400 Steps) Standard causal language modeling on the concatenated `(prompt + correct_label)` text. This phase teaches the model the basic structure of the task. - `max_steps`: 400 - `per_device_train_batch_size`: 16 - `learning_rate`: 2e-4 - `lr_scheduler_type`: cosine - `warmup_steps`: 100 ### Phase B: Direct Preference Optimization (200 Steps) DPO (Rafailov et al., 2023) directly optimizes the model to assign higher likelihood to chosen completions over rejected ones, without requiring a separate reward model. Beta controls how strongly the model is penalized for deviating from the base policy. - `max_steps`: 200 - `per_device_train_batch_size`: 16 - `learning_rate`: 5e-6 - `lr_scheduler_type`: cosine - `warmup_steps`: 40 - `beta`: 0.1 ### XGBoost Meta-Learner Ensemble An XGBoost Classifier (Chen and Guestrin, 2016) with 200 trees (max_depth=5, lr=0.05) is trained on `val[:300]`, a held-out set never seen during SFT or DPO. The feature vector combines raw technical features with LLM-derived probability signals: **Raw Technical Features:** - RSI(14), 1-week return %, 1-month return %, 20-day volatility, P/E ratio, ROE % - `rsi_oversold` (binary, RSI < 30), `rsi_overbought` (binary, RSI > 70) **LLM-Derived Features:** - `p_buy`, `p_sell`, `p_hold` - softmax probabilities from the fine-tuned LLM - `llm_confidence` = max(p_buy, p_sell, p_hold) - `buy_vs_hold_margin` = raw logit b - h - `sell_vs_hold_margin` = raw logit s - h Final prediction: **60% calibrated LLM + 40% XGBoost**. ### Temperature Calibration LLM logits are calibrated using Temperature Scaling (Guo et al., 2017) before ensemble combination. The optimal temperature T is found by minimizing Negative Log-Likelihood over `val[:300]` using scipy's bounded scalar optimizer in the range [0.1, 10.0]. T > 1 softens overconfident predictions; T < 1 sharpens underconfident ones. --- ## References - Hu, E. J., et al. (2021). **LoRA: Low-Rank Adaptation of Large Language Models.** arXiv:2106.09685 - Dettmers, T., et al. (2023). **QLoRA: Efficient Finetuning of Quantized LLMs.** arXiv:2305.14314 - Rafailov, R., et al. (2023). **Direct Preference Optimization: Your Language Model is Secretly a Reward Model.** arXiv:2305.18290 - Guo, C., et al. (2017). **On Calibration of Modern Neural Networks.** ICML 2017. arXiv:1706.04599 - Chen, T., and Guestrin, C. (2016). **XGBoost: A Scalable Tree Boosting System.** KDD 2016. arXiv:1603.02754 --- ## Disclaimer This model is intended for educational and quantitative research purposes only. It does not constitute financial advice. All trading decisions made using the outputs of this model are made at the user's own risk. The model has no access to real-time data, live order books, or macro-economic news events unless explicitly included in the prompt.