Update Weights

.gitattributes

@@ -1,36 +1,48 @@
 *.7z filter=lfs diff=lfs merge=lfs -text
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
 *.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
 *.ftz filter=lfs diff=lfs merge=lfs -text
 *.gz filter=lfs diff=lfs merge=lfs -text
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text
 *.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
 *.model filter=lfs diff=lfs merge=lfs -text
 *.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
 *.onnx filter=lfs diff=lfs merge=lfs -text
 *.ot filter=lfs diff=lfs merge=lfs -text
 *.parquet filter=lfs diff=lfs merge=lfs -text
 *.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
 *.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
 *.xz filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.gguf* filter=lfs diff=lfs merge=lfs -text
+*.ggml filter=lfs diff=lfs merge=lfs -text
+*.llamafile* filter=lfs diff=lfs merge=lfs -text
+*.pt2 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 *.png filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,55 @@
+Qwen RESEARCH LICENSE AGREEMENT
+
+Qwen RESEARCH LICENSE AGREEMENT Release Date: September 20, 2026
+
+By clicking to agree or by using or distributing any portion or element of the Qwen Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.
+
+1. Definitions
+    a. This Qwen RESEARCH LICENSE AGREEMENT (this "Agreement") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.
+    b. "We" (or "Us") shall mean Hangzhou Tongyi Laboratory Technology Co., Ltd.
+    c. "You" (or "Your") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.
+    d. "Third Parties" shall mean individuals or legal entities that are not under common control with us or you.
+    e. "Qwen" shall mean the large language models, diffusion models, and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing distributed by us.
+    f. "Materials" shall mean, collectively, our proprietary Qwen and Documentation (and any portion thereof) made available under this Agreement.
+    g. "Source" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.
+    h. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+    i. "Non-Commercial" shall mean for research or evaluation purposes only.
+
+2. Grant of Rights
+    a. You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under our intellectual property or other rights owned by us embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials FOR NON-COMMERCIAL PURPOSES ONLY. 
+    b. You shall not use the Materials for any commercial purpose without obtaining a separate commercial license from us. If you wish to use the Materials commercially, you shall request a license from us at model-business@notice.qwencloud.com.
+
+3. Redistribution
+Subject to Section 2 (Grant of Rights), you may distribute copies or make the Materials, or derivative works thereof, available as part of a product or service that contains any of them, with or without modifications, and in Source or Object form, provided that you meet the following conditions:
+    a. You shall give any other recipients of the Materials or derivative works a copy of this Agreement;
+    b. You shall cause any modified files to carry prominent notices stating that you changed the files;
+    c. You shall retain in all copies of the Materials that you distribute the following attribution notices within a "Notice" text file distributed as a part of such copies: "Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) 2026 Hangzhou Tongyi Laboratory Technology Co., Ltd. All Rights Reserved."; and
+    d. You may add your own copyright statement to your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of your modifications, or for any such derivative works as a whole, provided your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.
+
+4. Rules of use
+    a. The Materials may be subject to export controls or restrictions in China, the United States or other countries or regions. You shall comply with applicable laws and regulations in your use of the Materials.
+    b. If you use the Materials or any outputs or results therefrom to create, train, fine-tune, or improve an AI model that is distributed or made available, you shall prominently display “Built with Qwen” or “Improved using Qwen” in the related product documentation.
+    c. You shall not use "Qwen" as the primary name or identifier of any derivative works or products; reasonable descriptive use (e.g., "fine-tuned from Qwen Image") is permitted.
+
+5. Intellectual Property
+    a. We retain ownership of all intellectual property rights in and to the Materials and derivatives made by or for us. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by you, you are and will be the owner of such derivative works and modifications.
+    b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of us, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials.
+    c. If you commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against us or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by you, then all licenses granted to you under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought.
+
+6. Disclaimer of Warranty and Limitation of Liability
+    a. We are not obligated to support, update, provide training for, or develop any further version of the Qwen Materials or to grant any license thereto.
+    b. THE MATERIALS ARE PROVIDED "AS IS" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM.
+    c. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO ANY DIRECT, OR INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, NO MATTER HOW IT’S CAUSED.
+    d. You will defend, indemnify and hold harmless us from and against any claim by any third party arising out of or related to your use or distribution of the Materials.
+
+7. Survival and Termination.
+    a. The term of this Agreement shall commence upon your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.
+    b. We may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you must delete and cease use of the Materials. Sections 6 and 8 shall survive the termination of this Agreement.
+
+8. Governing Law and Jurisdiction.
+    a. This Agreement and any dispute arising out of or relating to it will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.
+    b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement.
+
+9. Other Terms and Conditions.
+    a. Any arrangements, understandings, or agreements regarding the Material not stated herein are separate from and independent of the terms and conditions of this Agreement. You shall request a separate license from us, if you use the Materials in ways not expressly agreed to in this Agreement. 
+    b. We shall not be bound by any additional or different terms or conditions communicated by you unless expressly agreed.

Qwen-Image-2.1-Fun-Controlnet-Union.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65d6b66d734da9e7ff5ef04e7db3a133553a52a3f29a7fcb3e9cce8fa21dcfcd
+size 7550979904

README.md

@@ -1,5 +1,150 @@
 ---
 license: other
-license_name: qwen-research
 license_link: LICENSE
+license_name: qwen-research
+library_name: videox_fun
+tags:
+- controlnet
+- controlnet-union
+- text-to-image
+- image-to-image
+- image-inpainting
+tasks:
+- text-to-image-synthesis
 ---
+
+# Qwen-Image-2.1-Fun-Controlnet-Union
+
+[![Github](https://img.shields.io/badge/🎨%20Code-VideoX_Fun-blue)](https://github.com/aigc-apps/VideoX-Fun)
+
+## Overview
+
+Qwen-Image-2.1-Fun-Controlnet-Union is a **ControlNet-Union branch for [Qwen-Image 2.1](https://github.com/aigc-apps/VideoX-Fun)** (the flow-matching text-to-image DiT). A single checkpoint drives **8 structural control conditions** (Canny, Depth, Grayscale, HED, Lineart, MLSD, Pose, Scribble) *and* image inpainting, without per-condition weights. The checkpoint holds **only the control branch** (`control_img_in` plus 16 `control_blocks`, about 7.0 GB) and is loaded on top of the base Qwen-Image 2.1 transformer.
+
+## Model Card
+
+| Name | Description |
+|--|--|
+| Qwen-Image-2.1-Fun-Controlnet-Union.safetensors | ControlNet-Union branch weights for Qwen-Image 2.1. Contains only the control branch (`control_img_in` + 16 `control_blocks`, about 7.0 GB); loaded with `strict=False` on top of the base Qwen-Image 2.1 transformer. One checkpoint covers 8 control conditions and image inpainting. |
+
+## Model Features
+- **Union control over 8 conditions**: one checkpoint handles Canny, Depth, Grayscale, HED, Lineart, MLSD, Pose and Scribble control images for text-to-image generation — no per-condition checkpoint switching.
+- **Dense control injection**: the control branch attaches a skip to every 2nd of the 32 transformer blocks (`control_layers = [0, 2, 4, …, 30]`, 16 injection points). Each control skip is added back to the main branch through zero-gated `before_proj` / `after_proj` projections, giving tight structural adherence while the base model stays frozen.
+- **Control and inpainting share one branch**: the control input is widened to `control_in_dim = 129` — `control latents (64) | mask (1) | masked-image latents (64)`. For pure control the mask / masked-image channels are zero-padded; for inpainting the same branch re-draws the masked region from the prompt. The two can also be **combined** — a control image and a mask are fed together, so the re-drawn region follows both the prompt and the given structure.
+- **CFG-distilled fast sampling**: the standalone example scripts run with `guidance_scale = 1.0` (single forward pass per step, no classifier-free guidance needed).
+- `control_context_scale` scales every control skip before it is added to the main branch: `1.0` is the strongest control (used for all results below), lower values weaken the guidance, `0.0` switches the control branch off.
+- **Prompt-friendly**: write a prompt that describes the **whole target image**; the masked region is conveyed by the mask channel, not by the text. Detailed prompts give better stability.
+- Qwen-Image 2.1 encodes the prompt (and any condition image) with a **Qwen3-VL** text encoder + processor, and its VAE decodes to **RGBA**, so every preview is saved as PNG.
+
+## Supported control conditions
+
+| Condition | Control signal |
+|--|--|
+| Canny | Canny edge map |
+| Depth | Monocular depth map |
+| Grayscale | Grayscale (luminance) image |
+| HED | HED edge detection map |
+| Lineart | Line-art extraction |
+| MLSD | Line-segment detection map |
+| Pose | DWPose skeleton |
+| Scribble | Free-hand / sketch lines |
+
+Any ordinary RGB control image at the target canvas works; the model tolerates different line thickness, thresholds and crops.
+
+## Results
+
+All samples below are generated with `num_inference_steps = 40`, `control_context_scale = 1.0`, seed 43. In each column the top row is the control image, the bottom row is the output.
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr><td>Canny</td><td>Depth</td><td>Grayscale</td><td>HED</td><td>Lineart</td><td>MLSD</td><td>Pose</td><td>Scribble</td></tr>
+  <tr>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_2_00025069.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_5_00005389.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_1_00014291.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_6_00009388.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_4_00003435.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_4_00027496.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_4_00000931.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_13_00000494.png" width="100%"></td>
+  </tr>
+  <tr>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_2_00025069.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_5_00005389.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_1_00014291.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_6_00009388.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_4_00003435.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_4_00027496.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_4_00000931.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_13_00000494.png" width="100%"></td>
+  </tr>
+</table>
+
+### Inpainting (+ control)
+
+A masked region of the source image is re-drawn from the prompt while the rest of the frame is preserved. The mask image is **white where the content should be re-generated** and **black where it should be kept**. Because control and inpainting share the same branch, a control image (here a DWPose skeleton) is fed together with the mask, so the re-drawn region also follows the given pose.
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr><td>Source image</td><td>Mask</td><td>Pose control</td><td>Inpaint output</td></tr>
+  <tr>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/inpaint_source.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/inpaint_mask.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/inpaint_control.jpg" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/inpaint.png" width="100%"></td>
+  </tr>
+</table>
+
+## Inference
+Go to the VideoX-Fun repository for more details.
+
+Please clone the VideoX-Fun repository and create the required directories:
+
+```sh
+# Clone the code
+git clone https://github.com/aigc-apps/VideoX-Fun.git
+
+# Enter VideoX-Fun's directory
+cd VideoX-Fun
+
+# Create model directories
+mkdir -p models/Diffusion_Transformer
+```
+
+Then download the base Qwen-Image 2.1 model and this checkpoint into `models/Diffusion_Transformer`.
+
+```
+📦 models/
+├──  Diffusion_Transformer/
+│   ├── 📂 Qwen-Image-2.1/
+│   └── 📂 Qwen-Image-2.1-Fun-Controlnet-Union/
+│       └──  Qwen-Image-2.1-Fun-Controlnet-Union.safetensors
+```
+
+Then edit the settings at the top of `examples/qwenimage21_fun/predict_t2i_control.py` (or `predict_i2i_inpaint.py` for inpainting) and run it.
+
+```python
+model_name          = "models/Diffusion_Transformer/Qwen-Image-2.1"
+config_path         = "config/qwenimage21/qwenimage21_control.yaml"
+transformer_path    = "models/Diffusion_Transformer/Qwen-Image-2.1-Fun-Controlnet-Union/Qwen-Image-2.1-Fun-Controlnet-Union.safetensors"
+control_image       = "asset/pose.jpg"
+# inpaint only:
+inpaint_image       = "asset/8.png"
+mask_image          = "asset/mask.png"
+prompt              = "your prompt describing the whole target image"
+```
+
+```sh
+python examples/qwenimage21_fun/predict_t2i_control.py
+```
+
+Notes:
+- `config_path` **must** be `config/qwenimage21/qwenimage21_control.yaml`. It builds the control branch exactly as the checkpoint expects (`control_layers: [0, 2, 4, …, 30]`, `control_in_dim: 129`); a mismatched config silently drops or misplaces control weights and produces wrong outputs.
+- For pure control (no inpaint input) the pipeline zero-pads the mask / masked-image channels, so this inpaint-capable checkpoint still runs plain Canny/Depth/… control correctly.
+- The control checkpoint carries only the control branch; the base Qwen-Image 2.1 weights must be present in `model_name`.
+- `control_context_scale = 1.0` is the value the adapter expects; use lower values to loosen the structural constraint.
+- `sample_size` sets the output canvas (e.g. `[1728, 992]`); keep both sides as multiples of 16 so the control map is not distorted.
+- `use_kv_cache = True` caches the text / condition-image keys after the first denoising step for a speedup at fixed resolution.
+- Memory: the transformer plus the Qwen3-VL text encoder do not fit a single consumer GPU fully loaded; use `model_group_offload` (fastest) or `model_cpu_offload_and_qfloat8` on a single high-memory GPU.
+
+## License
+
+This model is a derivative of Qwen-Image 2.1 and is released under the [Qwen Research License](https://modelscope.cn/models/Qwen/Qwen-Image-2.1/file/view/master/LICENSE). Please read the license carefully before use.

README_zh.md

@@ -0,0 +1,151 @@
+---
+license: other
+license_link: LICENSE
+license_name: qwen-research
+library_name: videox_fun
+tags:
+- controlnet
+- controlnet-union
+- text-to-image
+- image-to-image
+- image-inpainting
+tasks:
+- text-to-image-synthesis
+---
+
+# Qwen-Image-2.1-Fun-Controlnet-Union
+
+[![Github](https://img.shields.io/badge/🎨%20Code-VideoX_Fun-blue)](https://github.com/aigc-apps/VideoX-Fun)
+
+## 概述
+
+Qwen-Image-2.1-Fun-Controlnet-Union 是面向 **[Qwen-Image 2.1](https://github.com/aigc-apps/VideoX-Fun)**（flow-matching 文生图 DiT）的 **ControlNet-Union 控制分支**。单个 checkpoint 即可驱动 **8 种结构控制条件**（Canny、深度、灰度、HED、Lineart、MLSD、姿态、涂鸦）*以及*图像修复，无需为每种条件单独准备权重。checkpoint **仅包含控制分支**（`control_img_in` 与 16 个 `control_blocks`，约 7.0 GB），需叠加加载到 Qwen-Image 2.1 基础 transformer 之上。
+
+## 模型文件说明
+
+| 文件名 | 说明 |
+|--|--|
+| Qwen-Image-2.1-Fun-Controlnet-Union.safetensors | Qwen-Image 2.1 的 ControlNet-Union 分支权重，仅包含控制分支（`control_img_in` + 16 个 `control_blocks`，约 7.0 GB），以 `strict=False` 叠加加载到 Qwen-Image 2.1 基础 transformer 之上。单个 checkpoint 覆盖 8 种控制条件及图像修复。 |
+
+## 模型特性
+- **一个 checkpoint 统管 8 种控制条件**：Canny、深度、灰度、HED、Lineart、MLSD、姿态、涂鸦控制图的文生图，无需为每种条件单独切换 checkpoint。
+- **更密集的控制注入**：控制分支在 32 个 transformer 块中每 2 层接入一个跳接（`control_layers = [0, 2, 4, …, 30]`，共 16 个注入点）。每条控制跳接都经零初始化的 `before_proj` / `after_proj` 门控投影后叠加回主分支，在基础模型保持冻结的同时实现严格的结构还原。
+- **控制与修复共用同一分支**：控制输入拓宽至 `control_in_dim = 129` —— `控制潜变量(64) | 掩码(1) | 掩码后图像潜变量(64)`。纯控制时掩码/掩码图像通道补零；修复时同一分支根据 prompt 重绘掩码区域。二者还可**叠加使用**——同时输入一张控制图与掩码，被重绘的区域会同时跟随 prompt 与给定的结构。
+- **CFG 蒸馏快速采样**：独立的示例脚本以 `guidance_scale = 1.0` 推理（每步仅一次前向计算，无需 classifier-free guidance）。
+- `control_context_scale` 在控制跳接叠加回主分支前对其进行缩放：`1.0` 为最强控制（下方所有结果均用此值），调低可减弱结构约束，`0.0` 则完全关闭控制分支。
+- **提示词写法**：prompt 应描述**整张目标图像**；被打掩码的区域由掩码通道表达，而非文字。提示词越详细，生成越稳定。
+- Qwen-Image 2.1 使用 **Qwen3-VL** 文本编码器 + processor 编码 prompt（及条件图），其 VAE 解码为 **RGBA**，因此所有预览图均以 PNG 保存。
+
+## 支持的控制条件
+
+| 条件 | 控制信号 |
+|--|--|
+| Canny | Canny 边缘图 |
+| Depth | 单目深度图 |
+| Grayscale | 灰度（亮度）图 |
+| HED | HED 边缘检测图 |
+| Lineart | 线稿提取图 |
+| MLSD | 直线段检测图 |
+| Pose | DWPose 人体骨架 |
+| Scribble | 手绘 / 草图线条 |
+
+任何与目标画布尺寸一致的普通 RGB 控制图均可使用；模型对不同线条粗细、阈值与裁剪都有很好的容忍度。
+
+## 生成效果
+
+以下所有样例均在 `num_inference_steps = 40`、`control_context_scale = 1.0`、随机种子 43 的条件下生成。每列中，上排为控制图，下排为生成结果。
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr><td>Canny</td><td>Depth</td><td>Grayscale</td><td>HED</td><td>Lineart</td><td>MLSD</td><td>Pose</td><td>Scribble</td></tr>
+  <tr>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_2_00025069.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_5_00005389.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_1_00014291.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_6_00009388.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_4_00003435.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_4_00027496.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_4_00000931.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/control_13_00000494.png" width="100%"></td>
+  </tr>
+  <tr>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_2_00025069.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_5_00005389.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_1_00014291.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_6_00009388.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_4_00003435.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_4_00027496.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_4_00000931.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/control_13_00000494.png" width="100%"></td>
+  </tr>
+</table>
+
+### 图像修复（+ 控制）
+
+源图中被打上掩码的区域会根据 prompt 重新绘制，画面其余部分原样保留。掩码图中，**需要重新生成的区域为白色**，**需要保留的区域为黑色**。由于控制与修复共用同一分支，这里在掩码之外还同时输入了一张控制图（DWPose 骨架），被重绘的区域因此也会跟随给定的姿态。
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr><td>源图</td><td>掩码</td><td>姿态控制</td><td>修复输出</td></tr>
+  <tr>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/inpaint_source.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/inpaint_mask.png" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/asset/inpaint_control.jpg" width="100%"></td>
+    <td><img src="https://huggingface.co/alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union/resolve/main/results/inpaint.png" width="100%"></td>
+  </tr>
+</table>
+
+## 推理
+
+更多细节请参见 VideoX-Fun 代码仓库。
+
+首先克隆 VideoX-Fun 仓库并创建模型目录：
+
+```sh
+# 克隆代码
+git clone https://github.com/aigc-apps/VideoX-Fun.git
+
+# 进入 VideoX-Fun 目录
+cd VideoX-Fun
+
+# 创建模型目录
+mkdir -p models/Diffusion_Transformer
+```
+
+随后将 Qwen-Image 2.1 基础模型与本 checkpoint 下载至 `models/Diffusion_Transformer` 下：
+
+```
+📦 models/
+├──  Diffusion_Transformer/
+│   ├── 📂 Qwen-Image-2.1/
+│   └── 📂 Qwen-Image-2.1-Fun-Controlnet-Union/
+│       └──  Qwen-Image-2.1-Fun-Controlnet-Union.safetensors
+```
+
+接着修改 `examples/qwenimage21_fun/predict_t2i_control.py`（图像修复用 `predict_i2i_inpaint.py`）顶部的配置项，然后运行：
+
+```python
+model_name          = "models/Diffusion_Transformer/Qwen-Image-2.1"
+config_path         = "config/qwenimage21/qwenimage21_control.yaml"
+transformer_path    = "models/Diffusion_Transformer/Qwen-Image-2.1-Fun-Controlnet-Union/Qwen-Image-2.1-Fun-Controlnet-Union.safetensors"
+control_image       = "asset/pose.jpg"
+# 仅修复任务需要：
+inpaint_image       = "asset/8.png"
+mask_image          = "asset/mask.png"
+prompt              = "描述整张目标图像的 prompt"
+```
+
+```sh
+python examples/qwenimage21_fun/predict_t2i_control.py
+```
+
+注意事项：
+- `config_path` **必须**是 `config/qwenimage21/qwenimage21_control.yaml`：只有它能按 checkpoint 期望的结构完整搭建控制分支（`control_layers: [0, 2, 4, …, 30]`、`control_in_dim: 129`）；配置不匹配会静默丢弃或错位加载控制权重，产出错误结果。
+- 纯控制（不做修复输入）时，pipeline 会自动把掩码 / 掩码图像通道补零，因此这个支持修复的 checkpoint 一样能正确跑普通的 Canny/深度/… 控制任务。
+- 控制 checkpoint 只含控制分支，`model_name` 目录中必须有 Qwen-Image 2.1 基础权重。
+- `control_context_scale = 1.0` 是适配器期望的取值；调低可减弱结构约束。
+- `sample_size` 设定输出画布（如 `[1728, 992]`）；请将宽高均保持为 32 的倍数，以免控制图被拉伸变形。
+- `use_kv_cache = True` 会在首个去噪步后缓存文本 / 条件图的 key/value，在固定分辨率下加速推理。
+- 显存：transformer 加 Qwen3-VL 文本编码器无法在单张消费级显卡上全量驻留；单卡大显存请使用 `model_group_offload`（速度最快）或 `model_cpu_offload_and_qfloat8`。
+
+## 许可证
+
+本模型为 Qwen-Image 2.1 的衍生模型，依据 [Qwen Research License](https://modelscope.cn/models/Qwen/Qwen-Image-2.1/file/view/master/LICENSE) 发布。使用前请仔细阅读许可条款。

configuration.json

@@ -0,0 +1 @@
+{"task":"text-to-image-synthesis"}