Clarify merge pipeline: custom scripted merge (mergekit-inspired), document instruct-source starting point

README.md

@@ -12,7 +12,6 @@ tags:
 - cpt
 - repair
 - depth-upscaling
-- mergekit
 - base-model
 pipeline_tag: image-text-to-text
 base_model:
@@ -25,45 +24,43 @@ This repository contains the selected **repaired base checkpoint** for our 80B-c
 
 ## Summary
 
-This release is a **repair / recovery continued-pretraining checkpoint**, not a final instruct model and not a posttrained agent model.
+This release is a **repair / recovery continued-pretraining (CPT) checkpoint**, not a final instruct model and not a posttrained agent model.
 
 The published checkpoint is:
 
 - `v7-20260415-234214/checkpoint-100`
 
-It was selected from several checkpoint-100 candidates after a short multimodal repair-CPT run intended to stabilize a depth-upscaled 80B-class merged model and recover coherent base-model behavior.
+selected from several checkpoint-100 candidates after a short multimodal repair-CPT run intended to stabilize a custom depth-upscaled 80B-class merged model and recover coherent base-model behavior.
 
-## Important note on the source model
+## Important note on the source checkpoint
 
-At construction time, the upstream **raw 27B base checkpoint was not available** in the environment we were working from. The available upstream starting point was the **Qwen 3.5 27B instruct checkpoint**.
+At construction time, the upstream **raw 27B base checkpoint was not available** to us in the environment we were working from. The practical upstream starting point was the **Qwen 3.5 27B instruct checkpoint**.
 
-That matters for interpretation:
+This matters for interpretation:
 
-- this 80B checkpoint should still be treated as a **repaired base candidate**, not as a clean continuation of a pristine raw-base lineage
-- some degradation or drift in instruction-format adherence, style, and exact response behavior is expected during repair CPT
-- additional **posttraining repair and multimodal posttraining** are expected and planned after this stage
+- this 80B artifact should be treated as a **repaired base candidate**, not as a clean continuation of a pristine raw-base lineage
+- some degradation or drift in instruction-format adherence, response style, and formatting behavior is expected during repair CPT
+- additional **posttraining repair and posttraining proper** are expected and planned as the next stage
 
-In other words: this release is the best current repaired base candidate from the available source path, not the end of the pipeline.
+So this release is intentionally framed as the best current repaired base candidate from the available source path, not the end of the pipeline.
 
 ## Construction pipeline
 
 The model was built through a staged pipeline:
 
-1. **Abliteration / de-alignment pass** on the available Qwen 3.5 27B instruct source checkpoint
-2. **End-to-end depth upscaling / merge** using `mergekit` passthrough stacking to produce an ~80B-class dense multimodal model
-3. **Short repair CPT** to stabilize the merged model, improve text-side recovery, and preserve functional multimodal behavior
+1. **Abliteration / de-alignment pass** on the available Qwen 3.5 27B instruct source checkpoint.
+2. **Custom scripted end-to-end depth upscaling / merge** producing an ~80B-class dense multimodal model. The conceptual template is mergekit-style passthrough depth stacking, but the actual published artifact was produced through our **own merge / graft pipeline** written in-house after mergekit-based tooling paths did not work cleanly for this specific VLM setup.
+3. **Short repair CPT** to stabilize the merged model, recover coherent text-side behavior, and preserve functional multimodal capability.
 
-Conceptually, the goal was:
+Conceptually, the goals were:
 
 - preserve the first-class VLM nature of the Qwen 3.5 family
-- produce a larger dense multimodal base candidate quickly
-- use a short repair pass to recover coherence before later posttraining stages
+- produce a larger dense multimodal base candidate using depth upscaling instead of training a new 80B-class model from scratch
+- then use a short repair CPT pass to recover coherence before moving into posttraining
 
 ## Merge / architecture rationale
 
-The 80B model was created by **depth upscaling** rather than training an 80B model from scratch. The underlying idea is that repeating the pretrained stack end-to-end can create a deeper network that still inherits useful circuits from the source model, after which a repair CPT pass can smooth the new transitions.
-
-This is not intended as a claim that the merged checkpoint is fully recovered immediately after merging. The repair pass is a required part of the process.
+Rather than training an 80B model from scratch, we chose **depth upscaling** from a smaller pretrained source. The rationale is that repeating the pretrained stack end-to-end produces a deeper network that still inherits useful pretrained structure, and a short repair CPT pass can then smooth the new transitions. The merged artifact is not expected to be strong immediately out of merge; the repair CPT is an intentional part of the process, not an optional cleanup step.
 
 ## Repair CPT setup
 
@@ -73,37 +70,38 @@ The selected checkpoint came from a short multimodal repair-CPT pilot using:
 - nodes: **4** (32 B200 GPUs total)
 - dtype: **bf16**
 - mode: **full-parameter training**
-- max length: **8192**
+- max sequence length: **8192**
 - save cadence: **every 100 steps**
+- gradient checkpointing: **enabled**
+
+The first multimodal pilot had to normalize the training data format before use, because the raw FineVision-derived parquet metadata did not cleanly flow through the trainer's dataset path.
 
 ### Repair data mix
 
-The pilot used a conservative mixture intended to recover general text behavior while keeping multimodal/document competence alive:
+The pilot used a conservative mixture intended to recover general text-side behavior while keeping multimodal / document / OCR competence alive:
 
-- **75% text backbone**: cached **C4/en**
-- **25% multimodal/doc mix**, normalized into a training-friendly format from FineVision-derived sources:
+- **75% text backbone**: cached **C4 / en**
+- **25% multimodal / doc mix**, normalized into a training-friendly format from FineVision-derived sources:
   - `olmOCR-mix-0225-documents`
   - `olmOCR-mix-0225-books`
   - `docvqa`
   - `arxivqa`
   - `ocrvqa`
 
-The multimodal data had to be normalized before use because the raw parquet metadata path was not directly robust for the repair run stack.
-
 ## Why checkpoint-100 was selected
 
-This was treated as **circuit repair CPT**, not as a long-horizon capability training run where the lowest scalar loss automatically wins.
+This was explicitly treated as **circuit repair CPT**, not as a long-horizon capability training run, so the lowest absolute loss is not the primary criterion.
 
-The most important selection criteria were:
+The checkpoint was chosen based on:
 
-1. recover coherent text generation
-2. avoid obvious merge-path regressions
-3. retain clearly functional document/image understanding
-4. keep the checkpoint early and conservative enough to reduce risk of overfitting to the repair mix
+1. coherent text generation
+2. absence of obvious merge-path regressions
+3. clearly functional document / image understanding in qualitative checks
+4. preference for an early, conservative checkpoint to reduce risk of overfitting to the repair data mix
 
 ## Tiny checkpoint-selection validation
 
-Only **small validation subsets** were used at this stage. These were used only to identify the best repair checkpoint and to confirm that the model was recoverable. They should **not** be interpreted as final benchmark claims.
+Only **small validation subsets** were used at this stage. These were run purely to identify the best repair checkpoint and to confirm that the model was recoverable. They should **not** be interpreted as final benchmark claims.
 
 ### Plain Transformers smoke
 
@@ -112,7 +110,7 @@ Both the raw merged 80B checkpoint and checkpoint-100 were verified to:
 - load in plain `transformers`
 - generate tokens successfully
 
-In a trivial prompt smoke test, the selected checkpoint generated a clean answer, while the raw merged checkpoint remained visibly degraded.
+In a trivial prompt smoke test, the selected checkpoint produced a clean coherent response, while the raw merged checkpoint remained visibly degraded.
 
 ### Tiny GSM8K subset (n=5)
 
@@ -126,20 +124,20 @@ The published checkpoint was chosen from the strongest candidates in this tiny s
 
 ### Tiny multimodal qualitative check
 
-A small document/OCR-style qualitative check showed that the selected checkpoint is clearly able to:
+A small document / OCR-style qualitative check showed that the selected checkpoint is clearly able to:
 
 - read document pages
 - extract structure and content
 - produce semantically faithful long-form document responses
 
-A simple exact-match metric used in that quick check understated the practical multimodal capability because the model frequently returned structured, explanatory, or partially reformatted transcriptions instead of exact verbatim text. Those small multimodal checks were therefore useful for **sanity checking**, but not for rigorous ranking.
+A simple exact-match metric used in that check understated the practical multimodal capability, because the model frequently returned structured, explanatory or partially reformatted transcriptions instead of exact verbatim text. So the small multimodal checks were useful as **sanity checks**, but not for rigorous ranking.
 
 ## What this release is for
 
 Use this model as a **base candidate** for the next stages of work:
 
-- stronger multimodal repair CPT if needed
-- posttraining repair of instruction behavior/style
+- stronger multimodal repair CPT, if needed
+- posttraining repair of instruction behavior / formatting / style
 - supervised posttraining
 - preference / constitutional alignment
 - downstream agent-style posttraining
@@ -148,19 +146,20 @@ Use this model as a **base candidate** for the next stages of work:
 
 This repository should **not** be treated as:
 
-- a final instruct model
+- a final instruction-tuned model
 - a finished OCR-specialized model
 - a final benchmarked public release with comprehensive evaluation
 
-It is the selected **post-CPT repaired base checkpoint** from the current pipeline.
+It is the selected **post-CPT repaired base checkpoint** from the current pipeline stage.
 
 ## Expected next phase
 
 The next stage is expected to focus on **posttraining**, with additional attention to:
 
-- cleaning up instruction behavior inherited from the source-path constraints
+- cleaning up instruction behavior inherited from the instruct-source starting point
 - improving multimodal answer style and exactness where needed
 - running broader, more task-appropriate evaluations beyond the tiny checkpoint-selection subsets used here
+- optional additional repair CPT with a stronger multimodal focus if evaluation indicates it is needed
 
 ## Provenance
 
@@ -168,4 +167,3 @@ Published from:
 
 - source checkpoint: `v7-20260415-234214/checkpoint-100`
 - repository: `artivus-ai/qwen-3.5-80b-post-cpt-base`
-