Upload README.md with huggingface_hub

README.md

@@ -8,97 +8,164 @@ tags:
 - qwen3.5
 - multimodal
 - vlm
+- continued-pretraining
 - cpt
 - repair
+- depth-upscaling
+- mergekit
 - base-model
 pipeline_tag: image-text-to-text
 base_model:
-- Qwen/Qwen3.5-27B
+- Qwen/Qwen3.5-27B-Instruct
 ---
 
-# qwen-3.5-80b-post-cpt-base
+# Qwen 3.5 80B Post-CPT Base
 
-This repository contains the selected repaired base checkpoint for our 80B-class Qwen 3.5-derived VLM.
+This repository contains the selected **repaired base checkpoint** for our 80B-class Qwen 3.5-derived vision-language model.
 
-## What this is
+## Summary
 
-This model is **not** an instruct-tuned assistant and **not** a posttrained agent model.
-It is a **base-model recovery / repair CPT checkpoint** chosen after a short multimodal continued pretraining run intended to stabilize and repair a merged 80B base.
+This release is a **repair / recovery continued-pretraining checkpoint**, not a final instruct model and not a posttrained agent model.
 
-The selected checkpoint is:
+The published checkpoint is:
 
 - `v7-20260415-234214/checkpoint-100`
 
-chosen from several checkpoint-100 candidates because it showed the best tiny text-side recovery while preserving clearly functional document/image understanding.
+It was selected from several checkpoint-100 candidates after a short multimodal repair-CPT run intended to stabilize a depth-upscaled 80B-class merged model and recover coherent base-model behavior.
 
-## Training intent
+## Important note on the source model
 
-This run was treated as **circuit repair CPT**, not as a long horizon learning run.
-The goal was to recover a healthy base model after merge/assembly rather than continue broad capability training indefinitely.
+At construction time, the upstream **raw 27B base checkpoint was not available** in the environment we were working from. The available upstream starting point was the **Qwen 3.5 27B instruct checkpoint**.
 
-## Architecture notes
+That matters for interpretation:
 
-- Qwen 3.5-derived multimodal model
-- 80B-class repaired base
-- text backbone depth-expanded / merged from Qwen 3.5 lineage
-- vision tower retained as a first-class VLM component
+- this 80B checkpoint should still be treated as a **repaired base candidate**, not as a clean continuation of a pristine raw-base lineage
+- some degradation or drift in instruction-format adherence, style, and exact response behavior is expected during repair CPT
+- additional **posttraining repair and multimodal posttraining** are expected and planned after this stage
 
-## Data used in the repair run
+In other words: this release is the best current repaired base candidate from the available source path, not the end of the pipeline.
 
-The repair pilot used a mixture of:
+## Construction pipeline
 
-- text-side cached C4 data
-- normalized multimodal document / OCR / VQA data derived from FineVision-style parquet sources
+The model was built through a staged pipeline:
 
-The multimodal sources were normalized into a Swift/Transformers-friendly format before training.
+1. **Abliteration / de-alignment pass** on the available Qwen 3.5 27B instruct source checkpoint
+2. **End-to-end depth upscaling / merge** using `mergekit` passthrough stacking to produce an ~80B-class dense multimodal model
+3. **Short repair CPT** to stabilize the merged model, improve text-side recovery, and preserve functional multimodal behavior
 
-## Checkpoint selection summary
+Conceptually, the goal was:
 
-A tiny checkpoint sweep was run after training.
+- preserve the first-class VLM nature of the Qwen 3.5 family
+- produce a larger dense multimodal base candidate quickly
+- use a short repair pass to recover coherence before later posttraining stages
+
+## Merge / architecture rationale
+
+The 80B model was created by **depth upscaling** rather than training an 80B model from scratch. The underlying idea is that repeating the pretrained stack end-to-end can create a deeper network that still inherits useful circuits from the source model, after which a repair CPT pass can smooth the new transitions.
+
+This is not intended as a claim that the merged checkpoint is fully recovered immediately after merging. The repair pass is a required part of the process.
+
+## Repair CPT setup
+
+The selected checkpoint came from a short multimodal repair-CPT pilot using:
+
+- backend: **MS-Swift + DeepSpeed ZeRO-3**
+- nodes: **4** (32 B200 GPUs total)
+- dtype: **bf16**
+- mode: **full-parameter training**
+- max length: **8192**
+- save cadence: **every 100 steps**
+
+### Repair data mix
+
+The pilot used a conservative mixture intended to recover general text behavior while keeping multimodal/document competence alive:
+
+- **75% text backbone**: cached **C4/en**
+- **25% multimodal/doc mix**, normalized into a training-friendly format from FineVision-derived sources:
+  - `olmOCR-mix-0225-documents`
+  - `olmOCR-mix-0225-books`
+  - `docvqa`
+  - `arxivqa`
+  - `ocrvqa`
+
+The multimodal data had to be normalized before use because the raw parquet metadata path was not directly robust for the repair run stack.
+
+## Why checkpoint-100 was selected
+
+This was treated as **circuit repair CPT**, not as a long-horizon capability training run where the lowest scalar loss automatically wins.
+
+The most important selection criteria were:
+
+1. recover coherent text generation
+2. avoid obvious merge-path regressions
+3. retain clearly functional document/image understanding
+4. keep the checkpoint early and conservative enough to reduce risk of overfitting to the repair mix
+
+## Tiny checkpoint-selection validation
+
+Only **small validation subsets** were used at this stage. These were used only to identify the best repair checkpoint and to confirm that the model was recoverable. They should **not** be interpreted as final benchmark claims.
 
 ### Plain Transformers smoke
-Both the raw merged 80B and checkpoint-100 were verified to:
+
+Both the raw merged 80B checkpoint and checkpoint-100 were verified to:
 
 - load in plain `transformers`
-- generate tokens
+- generate tokens successfully
 
-The selected checkpoint produced a clean response to a trivial prompt, while the raw merged base remained visibly degraded.
+In a trivial prompt smoke test, the selected checkpoint generated a clean answer, while the raw merged checkpoint remained visibly degraded.
 
-### Tiny text eval (GSM8K subset, n=5)
-On a tiny direct `transformers` GSM8K slice:
+### Tiny GSM8K subset (n=5)
+
+A tiny direct `transformers` GSM8K slice was used for quick text-side checkpoint comparison:
 
 - `v6-20260415-234215/checkpoint-100`: **1 / 5**
 - `v7-20260415-164214/checkpoint-100`: **4 / 5**
 - `v7-20260415-234214/checkpoint-100`: **4 / 5**
 
-We selected `v7-20260415-234214/checkpoint-100` as the published repaired base.
+The published checkpoint was chosen from the strongest candidates in this tiny sweep.
 
 ### Tiny multimodal qualitative check
-A small document/OCR-style qualitative check showed that the model is clearly able to:
+
+A small document/OCR-style qualitative check showed that the selected checkpoint is clearly able to:
 
 - read document pages
-- extract content
-- describe / summarize page structure
+- extract structure and content
+- produce semantically faithful long-form document responses
+
+A simple exact-match metric used in that quick check understated the practical multimodal capability because the model frequently returned structured, explanatory, or partially reformatted transcriptions instead of exact verbatim text. Those small multimodal checks were therefore useful for **sanity checking**, but not for rigorous ranking.
+
+## What this release is for
+
+Use this model as a **base candidate** for the next stages of work:
+
+- stronger multimodal repair CPT if needed
+- posttraining repair of instruction behavior/style
+- supervised posttraining
+- preference / constitutional alignment
+- downstream agent-style posttraining
+
+## What this release is not
+
+This repository should **not** be treated as:
+
+- a final instruct model
+- a finished OCR-specialized model
+- a final benchmarked public release with comprehensive evaluation
 
-The exact-match metric used there was too strict for long-form document outputs, so those scores understate the practical multimodal capability. The model often returned semantically faithful but more structured / narrated responses instead of exact verbatim transcription.
+It is the selected **post-CPT repaired base checkpoint** from the current pipeline.
 
-## Recommended usage
+## Expected next phase
 
-Use this checkpoint as a **base model** for:
+The next stage is expected to focus on **posttraining**, with additional attention to:
 
-- further CPT repair
-- stronger multimodal retention training
-- later supervised / posttraining work
-- agent / tool / assistant posttraining stages
+- cleaning up instruction behavior inherited from the source-path constraints
+- improving multimodal answer style and exactness where needed
+- running broader, more task-appropriate evaluations beyond the tiny checkpoint-selection subsets used here
 
-## Caveats
+## Provenance
 
-- This is a repaired **base** model, not an instruction model.
-- Multimodal exact OCR-style faithfulness likely still benefits from additional targeted CPT or posttraining.
-- The evaluation described above was intentionally small and fast, used only for checkpoint selection, not for final benchmark claims.
+Published from:
 
-## Next planned work
+- source checkpoint: `v7-20260415-234214/checkpoint-100`
+- repository: `artivus-ai/qwen-3.5-80b-post-cpt-base`
 
-- stronger multimodal-focused recovery CPT if needed
-- publish later posttrained checkpoints separately
-- further VQA / doc QA / OCR evaluations with more task-appropriate metrics