MSweetbread
/

qwen2.5-1.5b-chess-qlora

@@ -1,6 +1,6 @@
 ---
 library_name: transformers
-tags: [chess, fine-tuned, lora, fen, uci]
 base_model: Qwen/Qwen2.5-1.5B
 ---
@@ -14,14 +14,14 @@ For this assignment, only a free tier of Google Colab was allowed to train your
 Additionally, the use of a pre-trained chess LLM was not permitted.
 ## Model Summary
-A LoRA fine-tuned version of the Qwen2.5-1.5B LLM for chess move prediction.
 Given a board state in the Forsyth-Edwards Notation (FEN) notation, the model outputs a move in the Universal Chess Interface (UCI) format.
 ## Model Architecture:
 - **Base model:** Qwen2.5-1.5B (Causal LM)
 - **Architecture:** Transformer with RoPE, SwiGLU, RMSNorm, Attention QKV bias and tied word embeddings
 - **Total parameters:** 1.54B (1.31B non-embedding)
-- **Trainable LoRA parameters:** 4,358,144 (0.49% of total)
 - **Layers:** 28 | **Context length:** 32,768 tokens
 ## Training Data
@@ -36,7 +36,7 @@ Lichess entries were filtered to engine depth ≥ 16 to ensure high-quality
 move annotations, at the cost of reduced dataset size.
 ## Training Procedure
-- **Method:** LoRA fine-tuning (PEFT)
 - **Training samples:** 227,134
 - **Effective batch size:** 64
 - **Training steps:** 7,098 (2 epochs)
@@ -61,7 +61,7 @@ print(tokenizer.decode(output[0], skip_special_tokens=True))
 ## Limitations
 - Training loss of 1.089 suggests the model is not highly confident in its
   predictions and will produce suboptimal moves in many positions.
-- LoRA at 0.49% of parameters means only a small fraction of the model was
   updated; chess knowledge is partially constrained by the base LLM's pretraining.
 - Training data consists primarily of high-level Lichess games (depth ≥ 16),
   meaning the model is tuned on strong engine moves and may struggle with

 ---
 library_name: transformers
+tags: [chess, fine-tuned, qlora, fen, uci]
 base_model: Qwen/Qwen2.5-1.5B
 ---
 Additionally, the use of a pre-trained chess LLM was not permitted.
 ## Model Summary
+A QLoRA fine-tuned version of the Qwen2.5-1.5B LLM for chess move prediction.
 Given a board state in the Forsyth-Edwards Notation (FEN) notation, the model outputs a move in the Universal Chess Interface (UCI) format.
 ## Model Architecture:
 - **Base model:** Qwen2.5-1.5B (Causal LM)
 - **Architecture:** Transformer with RoPE, SwiGLU, RMSNorm, Attention QKV bias and tied word embeddings
 - **Total parameters:** 1.54B (1.31B non-embedding)
+- **Trainable QLoRA parameters:** 4,358,144 (0.49% of total)
 - **Layers:** 28 | **Context length:** 32,768 tokens
 ## Training Data
 move annotations, at the cost of reduced dataset size.
 ## Training Procedure
+- **Method:** QLoRA fine-tuning (PEFT)
 - **Training samples:** 227,134
 - **Effective batch size:** 64
 - **Training steps:** 7,098 (2 epochs)
 ## Limitations
 - Training loss of 1.089 suggests the model is not highly confident in its
   predictions and will produce suboptimal moves in many positions.
+- QLoRA at 0.49% of parameters means only a small fraction of the model was
   updated; chess knowledge is partially constrained by the base LLM's pretraining.
 - Training data consists primarily of high-level Lichess games (depth ≥ 16),
   meaning the model is tuned on strong engine moves and may struggle with