Create README.md

README.md

@@ -0,0 +1,253 @@
+---
+license: apache-2.0
+base_model:
+- Qwen/Qwen2.5-0.5B-Instruct
+datasets:
+- agentlans/common-crawl-sample
+- bigcode/the-stack-smol-xl
+- rombodawg/Everything_Instruct
+tags:
+- draft
+- speculative-decoding
+---
+
+A `0.8B` parameter draft (speculative decoding) model for use with [command-a-03-2025](https://huggingface.co/CohereLabs/c4ai-command-a-03-2025).
+
+See [command-a-03-2025-DRAFT-0.8B-v3.0-GGUF](https://huggingface.co/jukofyork/command-a-03-2025-DRAFT-0.8B-v3.0-GGUF) for the models in `gguf` format for use with `llama.cpp`.
+
+
+---
+
+# Extending the context above 32k
+
+The current `config.json` is set for context length up to 32k tokens. Add the `"rope_scaling"` section to `config.json` to enable [YaRN](https://arxiv.org/abs/2309.00071), eg:
+
+## To extend the context to 64k:
+
+```json
+  "max_position_embeddings": 65536,
+  ...
+  "rope_scaling": {
+    "factor": 2.0,
+    "original_max_position_embeddings": 32768,
+    "type": "yarn"
+  },
+```
+
+## To extend the context to 128k:
+
+```json
+  "max_position_embeddings": 131072,
+  ...
+  "rope_scaling": {
+    "factor": 4.0,
+    "original_max_position_embeddings": 32768,
+    "type": "yarn"
+  },
+```
+
+**NOTE**: Because `llama.cpp` uses "static-YaRN" the scaling factor remains constant regardless of input length! Only add the `rope_scaling` configuration when processing long contexts is required...
+
+---
+
+# How this model was created
+
+## 1. The initial model was created from [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) using [transplant-vocab](https://github.com/jukofyork/transplant-vocab):
+
+```sh
+> python ./transplant_vocab.py \
+    ./Qwen2.5-0.5B-Instruct \
+    ./command-a-03-2025 \
+    ./command-a-03-2025-DRAFT-0.8B-UNTRAINED \
+    --override "<PAD>" "<|endoftext|>" \
+    --override "<UNK>" "<|endoftext|>" \
+    --override "<CLS>" "<|endoftext|>" \
+    --override "<SEP>" "<|endoftext|>" \
+    --override "<MASK_TOKEN>" "<|endoftext|>" \
+    --override "<BOS_TOKEN>" "<|endoftext|>" \
+    --override "<EOS_TOKEN>" "<|endoftext|>" \
+    --override "<EOP_TOKEN>" "<|endoftext|>" \
+    --override "<|START_OF_TURN_TOKEN|>" "<|im_start|>" \
+    --override "<|END_OF_TURN_TOKEN|>" "<|im_end|>" \
+    --override "<|YES_TOKEN|>" "<|endoftext|>" \
+    --override "<|NO_TOKEN|>" "<|endoftext|>" \
+    --override "<|GOOD_TOKEN|>" "<|endoftext|>" \
+    --override "<|BAD_TOKEN|>" "<|endoftext|>" \
+    --override "<|USER_TOKEN|>" "user\n" \
+    --override "<|CHATBOT_TOKEN|>" "assistant\n" \
+    --override "<|SYSTEM_TOKEN|>" "system\n" \
+    --override "<|START_THINKING|>" "<think>" \
+    --override "<|END_THINKING|>" "</think>" \
+    --override "<|START_RESPONSE|>" "<|endoftext|>" \
+    --override "<|END_RESPONSE|>" "<|endoftext|>" \
+    --override "<|START_ACTION|>" "<tool_call>" \
+    --override "<|END_ACTION|>" "</tool_call>" \
+    --override "<|START_TOOL_RESULT|>" "<tool_response>" \
+    --override "<|END_TOOL_RESULT|>" "</tool_response>" \
+    --override "<|BEGINNING_OF_PREFIX_FIM_TOKEN|>" "<|fim_prefix|>" \
+    --override "<|BEGINNING_OF_MIDDLE_FIM_TOKEN|>" "<|fim_middle|>" \
+    --override "<|BEGINNING_OF_SUFFIX_FIM_TOKEN|>" "<|fim_suffix|>" \
+    --override "<|END_OF_MIDDLE_FIM_TOKEN|>" "<|fim_middle|>"
+
+Loading config from 'Qwen2.5-0.5B-Instruct'... Done.
+Loading config from 'command-a-03-2025'... Done.
+Loading tokenizer from 'Qwen2.5-0.5B-Instruct'... Done.
+Loading tokenizer from 'command-a-03-2025'... Done.
+Loading model from 'Qwen2.5-0.5B-Instruct'... Done.
+
+Input model configuration:
+- Target vocabulary size    : 256000 (used = 255033, unused = 967)
+- Donor vocabulary size     : 151936
+- Donor num layers          : 24 (tied embeddings = True)
+- Donor hidden size         : 896
+- Donor attention heads     : 14
+- Donor intermediate size   : 4864 (ratio = 1:5.4)
+- Donor total parameters    : 494032768 (0.49B)
+-- Embedding parameters     : 136134656 (0.14B)
+-- Non-embedding parameters : 357898112 (0.36B)
+
+Processing 3 automatic token overrides:
+✔ 'bos_token_id' : 5 '<BOS_TOKEN>' → [151643] '<|endoftext|>'
+✔ 'eos_token_id' : 255001 '<|END_OF_TURN_TOKEN|>' → [151645] '<|im_end|>'
+✔ 'pad_token_id' : 0 '<PAD>' → [151643] '<|endoftext|>'
+
+Processing 29 manual token overrides:
+✔      0 : '<PAD>' → [151643] '<|endoftext|>'
+✔      1 : '<UNK>' → [151643] '<|endoftext|>'
+✔      2 : '<CLS>' → [151643] '<|endoftext|>'
+✔      3 : '<SEP>' → [151643] '<|endoftext|>'
+✔      4 : '<MASK_TOKEN>' → [151643] '<|endoftext|>'
+✔      5 : '<BOS_TOKEN>' → [151643] '<|endoftext|>'
+✔      6 : '<EOS_TOKEN>' → [151643] '<|endoftext|>'
+✔      7 : '<EOP_TOKEN>' → [151643] '<|endoftext|>'
+✔ 255000 : '<|START_OF_TURN_TOKEN|>' → [151644] '<|im_start|>'
+✔ 255001 : '<|END_OF_TURN_TOKEN|>' → [151645] '<|im_end|>'
+✔ 255002 : '<|YES_TOKEN|>' → [151643] '<|endoftext|>'
+✔ 255003 : '<|NO_TOKEN|>' → [151643] '<|endoftext|>'
+✔ 255004 : '<|GOOD_TOKEN|>' → [151643] '<|endoftext|>'
+✔ 255005 : '<|BAD_TOKEN|>' → [151643] '<|endoftext|>'
+✔ 255006 : '<|USER_TOKEN|>' → [872, 198] 'user\n'
+✔ 255007 : '<|CHATBOT_TOKEN|>' → [77091, 198] 'assistant\n'
+✔ 255008 : '<|SYSTEM_TOKEN|>' → [8948, 198] 'system\n'
+✔ 255019 : '<|START_THINKING|>' → [13708, 766, 29] '<think>'
+✔ 255020 : '<|END_THINKING|>' → [522, 26865, 29] '</think>'
+✔ 255021 : '<|START_RESPONSE|>' → [151643] '<|endoftext|>'
+✔ 255022 : '<|END_RESPONSE|>' → [151643] '<|endoftext|>'
+✔ 255023 : '<|START_ACTION|>' → [151657] '<tool_call>'
+✔ 255024 : '<|END_ACTION|>' → [151658] '</tool_call>'
+✔ 255025 : '<|START_TOOL_RESULT|>' → [27, 14172, 9655, 29] '<tool_response>'
+✔ 255026 : '<|END_TOOL_RESULT|>' → [522, 14172, 9655, 29] '</tool_response>'
+✔ 255029 : '<|BEGINNING_OF_PREFIX_FIM_TOKEN|>' → [151659] '<|fim_prefix|>'
+✔ 255030 : '<|BEGINNING_OF_MIDDLE_FIM_TOKEN|>' → [151660] '<|fim_middle|>'
+✔ 255031 : '<|BEGINNING_OF_SUFFIX_FIM_TOKEN|>' → [151661] '<|fim_suffix|>'
+✔ 255032 : '<|END_OF_MIDDLE_FIM_TOKEN|>' → [151660] '<|fim_middle|>'
+
+NOTE: Using an "untied" copy of 'embed_tokens.weight' as new 'lm_head.weight' tensor...
+
+Transplanting tokens: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 255033/255033 [01:58<00:00, 2145.75token/s]
+
+Transplant mappings:
+- 1 to 1  : 87077 (34%)
+- 2 to 1  : 117015 (46%)
+- 3 to 1  : 33833 (13%)
+- 4 to 1  : 10325 (4%)
+- 5 to 1  : 3415 (1.3%)
+- 6 to 1  : 1486 (0.58%)
+- 7 to 1  : 761 (0.3%)
+- 8 to 1  : 440 (0.17%)
+- 9 to 1  : 302 (0.12%)
+- 10 to 1 : 177 (0.069%)
+- 11 to 1 : 88 (0.035%)
+- 12 to 1 : 47 (0.018%)
+- 13 to 1 : 28 (0.011%)
+- 14 to 1 : 15 (0.0059%)
+- 15 to 1 : 8 (0.0031%)
+- 16 to 1 : 6 (0.0024%)
+- 17 to 1 : 1 (0.00039%)
+- 18 to 1 : 3 (0.0012%)
+- 19 to 1 : 1 (0.00039%)
+- 21 to 1 : 2 (0.00078%)
+- 36 to 1 : 1 (0.00039%)
+- 37 to 1 : 1 (0.00039%)
+- 39 to 1 : 1 (0.00039%)
+
+Head initialized with:
+- Copies : 87077 (34%)
+- Means  : 167956 (66%)
+- Zeros  : 967 (0.38%)
+
+Output model configuration:
+- Output vocabulary size    : 256000
+- Output num layers         : 24 (tied embeddings = False)
+- Output hidden size        : 896
+- Output attention heads    : 14
+- Output intermediate size  : 4864 (ratio = 1:5.4)
+- Output total parameters   : 816650112 (0.82B)
+-- Embedding parameters     : 458752000 (0.46B)
+-- Non-embedding parameters : 357898112 (0.36B)
+
+Saving model and tokenizer to 'command-a-03-2025-DRAFT-0.8B-UNTRAINED' folder
+
+Patching 'torch_dtype' in 'command-a-03-2025-DRAFT-0.8B-UNTRAINED/config.json' based on actual saved tensors
+- Updated 'torch_dtype' to 'bfloat16' based on actual tensor dtype
+
+Operation completed successfully (ignore any 'segmentation fault' that follows!!!)
+```
+
+## 2. The following datasets were used to create a fine-tuning dataset of ~2.4B tokens:
+
+- [agentlans/common-crawl-sample](https://huggingface.co/datasets/agentlans/common-crawl-sample)
+- [bigcode/the-stack-smol-xl](https://huggingface.co/datasets/bigcode/the-stack-smol-xl)
+- [rombodawg/Everything_Instruct](https://huggingface.co/datasets/rombodawg/Everything_Instruct) (NOTE: `output` field only)
+
+formatted just between `<|END_OF_TURN_TOKEN|>` tags.
+
+## 3. The model was then trained using [qlora-pipe-lite](https://github.com/jukofyork/qlora-pipe-lite) for 1 epoch with a batch size of 60 and a sequence length of 32k (~2M tokens per step):
+
+```toml
+# ==============================
+# MODEL AND OUTPUT CONFIGURATION
+# ==============================
+
+model_dir = 'models/command-a-03-2025-DRAFT-0.8B-UNTRAINED'
+output_dir = 'finetuned'
+
+# ===========================
+# TRAINING TYPE CONFIGURATION
+# ===========================
+
+full_fine_tune = true
+
+# =======================
+# OPTIMIZER CONFIGURATION
+# =======================
+
+lr = 5e-5
+
+# ======================
+# TRAINING CONFIGURATION
+# ======================
+
+sequence_len = 32768
+
+gradient_accumulation_steps = 10  # 10×6 = batch size 60, 10×6×32768 = ~2M tokens per step
+
+# =====================
+# DATASET CONFIGURATION
+# =====================
+
+drop_tails = true
+
+[[datasets]]
+dataset_path = 'datasets/common-crawl-sample/*.json'
+
+[[datasets]]
+dataset_path = 'datasets/the-stack-smol-xl/*.jsonl'
+
+[[datasets]]
+dataset_path = 'datasets/rombodawg-Everything-Instruct/*.json'
+```
+
+I used six `RTX A6000` GPUs over three nodes and hence the `60` batch size (`6 x 10 gradient accumulation steps = 60`).
+
+![image](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/SrGgAcVV45bxBAQxRnZit.png)