| ---
|
| license: apache-2.0
|
| base_model: stabilityai/stable-diffusion-xl-base-1.0
|
| tags:
|
| - stable-diffusion-xl
|
| - sdxl
|
| - text-to-image
|
| - diffusers
|
| - lora
|
| - control
|
| - controlnet
|
| - control-lora
|
| - brightness
|
| - grayscale
|
| - template:sd-lora
|
| widget:
|
| - text: "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"
|
| output:
|
| url: "https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/example.png"
|
| inference: true
|
| ---
|
|
|
| # ControlNet LoRA SDXL - Brightness Control (10k @ 1024×1024)
|
|
|
| A Control LoRA model trained on Stable Diffusion XL to control image generation through brightness/grayscale information. This model uses **LoRA (Low-Rank Adaptation)** combined with ControlNet architecture for efficient control, providing an **ultra-lightweight alternative** to full ControlNet with excellent pattern preservation.
|
|
|
| ## Model Description
|
|
|
| This Control LoRA enables brightness-based conditioning for SDXL image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.
|
|
|
| ### Key Features:
|
|
|
| - 🎨 **Excellent brightness and pattern control** across multiple scales (0.5-2.0)
|
| - 🚀 **196x smaller than full ControlNet**: ~24MB vs ~4.7GB
|
| - ⚡ **Ultra-fast loading**: LoRA weights load in <1 second
|
| - 💡 **Flexible scale control**: Adjustable conditioning scale from 0.5 to 2.0+
|
| - 🔄 **Compatible with ControlLoRA v3**: Uses the efficient ControlLoRA v3 architecture
|
| - 📦 **Minimal storage**: All checkpoints + final model = ~120MB total
|
| - 🖼️ **Native SDXL resolution**: Trained at 1024×1024
|
|
|
| ### Intended Uses:
|
|
|
| - **Artistic QR code generation** (scale 1.0-1.5 recommended)
|
| - Image recoloring and colorization
|
| - Lighting control in text-to-image generation
|
| - Brightness-based pattern integration
|
| - Watermark and subtle pattern embedding
|
| - Photo enhancement and stylization
|
|
|
| ## Training Details
|
|
|
| ### Training Data
|
|
|
| Trained on 10,000 samples from `latentcat/grayscale_image_aesthetic_3M`:
|
| - High-quality aesthetic images
|
| - Paired with grayscale/brightness versions
|
| - Native resolution: 1024×1024 (SDXL native)
|
|
|
| ### Training Configuration
|
|
|
| | Parameter | Value |
|
| |-----------|-------|
|
| | **Base Model** | `stabilityai/stable-diffusion-xl-base-1.0` |
|
| | **Architecture** | ControlLoRA v3 (~7M trainable parameters) |
|
| | **LoRA Rank** | 16 |
|
| | **Extra Conv Rank** | 64 (conv_in layer) |
|
| | **Training Resolution** | 1024×1024 |
|
| | **Training Steps** | 313 (1 epoch) |
|
| | **Batch Size** | 8 per device |
|
| | **Gradient Accumulation** | 4 (effective batch: 32) |
|
| | **Learning Rate** | 1e-4 |
|
| | **Empty Prompts** | 10% (ControlLoRA v3 recommendation) |
|
| | **Mixed Precision** | BF16 |
|
| | **Hardware** | NVIDIA H100 80GB |
|
| | **Training Time** | ~21 minutes |
|
| | **Final Loss** | ~0.10-0.12 |
|
|
|
| ### Model Size Comparison
|
|
|
| | Model | Parameters | Size | Training | Resolution |
|
| |-------|-----------|------|----------|-----------|
|
| | **This Control LoRA** | ~7M | **~24MB** | 10k @ 1024 | 1024×1024 |
|
| | ControlNet (SDXL) | ~700M | 4.7GB | 100k @ 512 | 512×512 |
|
| | T2I Adapter (SDXL) | ~77M | 302MB | 100k @ 1024 | 1024×1024 |
|
| | Flux Control LoRA | ~7M | 25MB | 10k @ 512 | 512×512 |
|
|
|
| ## Usage
|
|
|
| ### Installation
|
|
|
| ```bash
|
| pip install diffusers transformers accelerate torch peft
|
| # Install ControlLoRA v3
|
| git clone https://github.com/HighCWu/control-lora-v3
|
| ```
|
|
|
| ### Basic Usage
|
|
|
| ```python
|
| import torch
|
| import sys
|
| sys.path.insert(0, '/path/to/control-lora-v3')
|
|
|
| from pipeline_sdxl import StableDiffusionXLControlLoraV3Pipeline
|
| from model import UNet2DConditionModelEx
|
| from PIL import Image
|
|
|
| # Load UNet with LoRA support
|
| unet = UNet2DConditionModelEx.from_pretrained(
|
| "stabilityai/stable-diffusion-xl-base-1.0",
|
| subfolder="unet",
|
| torch_dtype=torch.bfloat16,
|
| )
|
| unet = unet.add_extra_conditions(["brightness"])
|
|
|
| # Load SDXL Control LoRA pipeline
|
| pipe = StableDiffusionXLControlLoraV3Pipeline.from_pretrained(
|
| "stabilityai/stable-diffusion-xl-base-1.0",
|
| unet=unet,
|
| torch_dtype=torch.bfloat16,
|
| )
|
|
|
| # Load Control LoRA weights
|
| pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness")
|
| pipe.to("cuda")
|
|
|
| # Load grayscale/brightness control image
|
| control_image = Image.open("path/to/grayscale_image.png")
|
| control_image = control_image.resize((1024, 1024))
|
|
|
| # Generate image
|
| prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"
|
|
|
| image = pipe(
|
| prompt=prompt,
|
| image=control_image,
|
| num_inference_steps=30,
|
| guidance_scale=7.5,
|
| extra_condition_scale=1.0, # Controls conditioning strength
|
| height=1024,
|
| width=1024,
|
| ).images[0]
|
|
|
| image.save("output.png")
|
| ```
|
|
|
| ### Adjusting Control Strength
|
|
|
| The `extra_condition_scale` parameter controls how strongly the brightness map influences generation:
|
|
|
| ```python
|
| # Subtle control (scale 0.5-0.7)
|
| image = pipe(
|
| prompt=prompt,
|
| image=control_image,
|
| extra_condition_scale=0.5,
|
| ...
|
| ).images[0]
|
|
|
| # Balanced control (scale 1.0-1.5) - Recommended for artistic QR codes
|
| image = pipe(
|
| prompt=prompt,
|
| image=control_image,
|
| extra_condition_scale=1.0,
|
| ...
|
| ).images[0]
|
|
|
| # Strong control (scale 1.5-2.0)
|
| image = pipe(
|
| prompt=prompt,
|
| image=control_image,
|
| extra_condition_scale=1.5,
|
| ...
|
| ).images[0]
|
| ```
|
|
|
| ### Artistic QR Code Generation
|
|
|
| ```python
|
| import qrcode
|
| from PIL import Image
|
|
|
| # Generate QR code
|
| qr = qrcode.QRCode(
|
| version=1,
|
| error_correction=qrcode.constants.ERROR_CORRECT_H,
|
| box_size=10,
|
| border=4
|
| )
|
| qr.add_data("https://your-url.com")
|
| qr.make(fit=True)
|
|
|
| qr_image = qr.make_image(fill_color="black", back_color="white")
|
| qr_image = qr_image.resize((1024, 1024), Image.LANCZOS).convert("RGB")
|
|
|
| # Generate artistic QR code (scale 1.0-1.5 works best)
|
| image = pipe(
|
| prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
|
| image=qr_image,
|
| num_inference_steps=30,
|
| guidance_scale=7.5,
|
| extra_condition_scale=1.0,
|
| height=1024,
|
| width=1024,
|
| ).images[0]
|
|
|
| image.save("artistic_qr.png")
|
| ```
|
|
|
| ### Using Different Checkpoints
|
|
|
| The model includes intermediate checkpoints from throughout training:
|
|
|
| ```python
|
| # Early checkpoint (25% - 2,500 samples)
|
| pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
|
| adapter_name="brightness",
|
| subfolder="checkpoint-78")
|
|
|
| # Mid checkpoint (50% - 5,000 samples)
|
| pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
|
| adapter_name="brightness",
|
| subfolder="checkpoint-156")
|
|
|
| # Late checkpoint (75% - 7,500 samples)
|
| pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
|
| adapter_name="brightness",
|
| subfolder="checkpoint-234")
|
|
|
| # Near-final checkpoint (99% - 9,984 samples)
|
| pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
|
| adapter_name="brightness",
|
| subfolder="checkpoint-312")
|
|
|
| # Final model (10,000 samples, main branch - recommended)
|
| pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl-10k",
|
| adapter_name="brightness")
|
| ```
|
|
|
| ## Conditioning Scale Guide
|
|
|
| The `extra_condition_scale` parameter controls how strongly the brightness map influences generation:
|
|
|
| ### Recommended Scale Ranges
|
|
|
| | Scale | Behavior | Best For |
|
| |-------|----------|----------|
|
| | **0.5-0.7** | Subtle artistic integration with hints of pattern | Natural images, soft lighting hints |
|
| | **0.7-1.0** | Light control - visible structure with artistic freedom | Artistic images, creative reinterpretation |
|
| | **1.0-1.5** | **🔥 Balanced control** | **Artistic QR codes, watermarks** (recommended) |
|
| | **1.5-2.0** | Strong control - clear patterns with artistic overlay | Geometric patterns, structured designs |
|
| | **2.0+** | Maximum control - dominant patterns | Strong brightness maps, technical applications |
|
|
|
| ## Performance Comparison
|
|
|
| ### vs Full ControlNet (SDXL)
|
|
|
| | Metric | ControlNet (SDXL) | This Control LoRA | Advantage |
|
| |--------|-------------------|-------------------|-----------|
|
| | **Parameters** | ~700M | ~7M | **100x smaller** |
|
| | **Model Size** | 4.7GB | 24MB | **196x smaller** |
|
| | **Load Time** | ~5-10 seconds | **<1 second** | **10x faster loading** |
|
| | **Storage (w/ checkpoints)** | ~18.8GB | **~120MB** | **157x less storage** |
|
| | **Training Time** | ~3 hours | **21 minutes** | **8.5x faster** |
|
| | **Pattern Preservation @ Scale 1.0** | Excellent | **Excellent** | Comparable quality |
|
| | **Flexibility** | Fixed architecture | **Adjustable weights** | More versatile |
|
|
|
| ### vs T2I Adapter (SDXL)
|
|
|
| | Metric | T2I Adapter (SDXL) | This Control LoRA | Advantage |
|
| |--------|-------------------|-------------------|-----------|
|
| | **Parameters** | ~77M | ~7M | **11x smaller** |
|
| | **Model Size** | 302MB | 24MB | **12.6x smaller** |
|
| | **Training Samples** | 100k | 10k | **More efficient** |
|
| | **Architecture** | Separate adapter | **Integrated LoRA** | Simpler loading |
|
|
|
| ## Checkpoint Progression Analysis
|
|
|
| The model includes checkpoints from throughout training:
|
|
|
| 1. **checkpoint-78**: 25% complete (2,500 samples)
|
| 2. **checkpoint-156**: 50% complete (5,000 samples)
|
| 3. **checkpoint-234**: 75% complete (7,500 samples)
|
| 4. **checkpoint-312**: 99% complete (9,984 samples)
|
| 5. **Final model**: 100% complete (10,000 samples - main branch)
|
|
|
| ### Visual Comparison
|
|
|
| Each comparison shows QR input + all 7 conditioning scales (0.25, 0.5, 0.7, 0.75, 1.0, 1.25, 1.5) for a specific checkpoint:
|
|
|
| #### Checkpoint 78 (25% trained, 2,500 samples)
|
| 
|
|
|
| #### Checkpoint 156 (50% trained, 5,000 samples)
|
| 
|
|
|
| #### Checkpoint 234 (75% trained, 7,500 samples)
|
| 
|
|
|
| #### Checkpoint 312 (99% trained, 9,984 samples)
|
| 
|
|
|
| #### Final Model (100% trained, 10,000 samples) - Recommended
|
| 
|
|
|
| ### Key Observations
|
|
|
| All checkpoints show consistent, high-quality performance across scales. The progression analysis reveals:
|
|
|
| 1. **Early Checkpoint (78 steps, 2.5k samples)**:
|
| - Good pattern awareness, developing control
|
| - More artistic interpretation of prompts
|
| - Recommended scales: 0.7-1.2
|
|
|
| 2. **Mid Checkpoints (156-234 steps, 5k-7.5k samples)**:
|
| - Strong balance between control and creativity
|
| - Stable pattern preservation
|
| - Recommended scales: 0.8-1.5
|
|
|
| 3. **Final Model (313 steps, 10k samples)**:
|
| - Maximum control capability
|
| - Excellent pattern preservation at all scales
|
| - Recommended scales: 0.7-2.0
|
|
|
| ### No Overfitting Observed
|
|
|
| Unlike larger models trained on 100k samples, this 10k Control LoRA shows **no signs of overfitting**:
|
| - Consistent improvement throughout training
|
| - Final checkpoint is recommended for production use
|
| - 10k samples appears optimal for LoRA-based control training
|
|
|
| ## When to Use This Model
|
|
|
| ### ✅ Use This Control LoRA When:
|
|
|
| - Creating **artistic QR codes** with SDXL quality (scale 1.0-1.5)
|
| - Need **minimal storage overhead** (<30MB per checkpoint)
|
| - Want **fast model loading** (<1 second)
|
| - Building **production applications** requiring small model sizes
|
| - Working with **SDXL** as base model
|
| - Require **flexible control strength** via extra_condition_scale
|
| - Need **multiple checkpoints** without massive storage (120MB total vs 18.8GB)
|
| - Training on **limited data** (10k samples sufficient)
|
|
|
| ### ⚠️ Consider Alternatives When:
|
|
|
| - Need **full ControlNet** features with extremely precise control
|
| - Working with existing **T2I Adapter** pipelines
|
| - Require **different control types** (pose, depth, etc.) - train separate LoRAs
|
|
|
| ## Limitations
|
|
|
| ### Current Limitations
|
|
|
| - **ControlLoRA v3 dependency**: Requires custom pipeline code (not in main diffusers yet)
|
| - **Grayscale conditioning only**: Trained specifically for brightness/grayscale control
|
| - **Single control type**: Only brightness, not other conditioning types
|
| - **Custom code required**: Need to include ControlLoRA v3 files
|
|
|
| ### Recommendations
|
|
|
| - For **SDXL generation**, use this Control LoRA
|
| - For **multiple control types**, train separate LoRAs and combine
|
| - Experiment with **scales 1.0-1.5** for most use cases
|
| - Use **final model** for best results
|
|
|
| ## Training Script
|
|
|
| ```bash
|
| accelerate launch --mixed_precision="bf16" train_sdxl.py \
|
| --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
|
| --dataset_name="<path_to_10k_dataset>" \
|
| --conditioning_image_column="conditioning_image" \
|
| --image_column="image" \
|
| --caption_column="text" \
|
| --output_dir="./controlnet-lora-brightness-sdxl-10k" \
|
| --mixed_precision="bf16" \
|
| --resolution=1024 \
|
| --learning_rate=1e-4 \
|
| --proportion_empty_prompts=0.1 \
|
| --rank=16 \
|
| --lora_adapter_name="brightness" \
|
| --extra_lora_rank_modules conv_in \
|
| --extra_lora_ranks 64 \
|
| --half_or_full_lora=half_skip_attn \
|
| --train_batch_size=8 \
|
| --num_train_epochs=1 \
|
| --gradient_accumulation_steps=4 \
|
| --gradient_checkpointing \
|
| --checkpointing_steps=78 \
|
| --validation_steps=78 \
|
| --validation_image="validation_qr.png" \
|
| --validation_prompt="a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" \
|
| --num_validation_images=4 \
|
| --seed=42 \
|
| --dataloader_num_workers=4 \
|
| --tracker_project_name="controlnet-lora-brightness-sdxl-10k" \
|
| --report_to="wandb" \
|
| --enable_xformers_memory_efficient_attention \
|
| --use_8bit_adam \
|
| --init_lora_weights="gaussian"
|
| ```
|
|
|
| ## Available Checkpoints
|
|
|
| All checkpoints are available in the main branch:
|
|
|
| - **Root directory**: Final model (10,000 samples, recommended)
|
| - **checkpoint-78/**: Early checkpoint (2,500 samples, 25% trained)
|
| - **checkpoint-156/**: Mid checkpoint (5,000 samples, 50% trained)
|
| - **checkpoint-234/**: Late checkpoint (7,500 samples, 75% trained)
|
| - **checkpoint-312/**: Near-final checkpoint (9,984 samples, 99% trained)
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @misc{controlnet-lora-brightness-sdxl,
|
| author = {Oysiyl},
|
| title = {ControlNet LoRA SDXL - Brightness Control (10k @ 1024×1024)},
|
| year = {2026},
|
| publisher = {HuggingFace},
|
| journal = {HuggingFace Model Hub},
|
| howpublished = {\url{https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl}}
|
| }
|
| ```
|
|
|
| ## Acknowledgments
|
|
|
| - Built with [🤗 Diffusers](https://github.com/huggingface/diffusers)
|
| - Base model: [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) by Stability AI
|
| - ControlLoRA v3: [control-lora-v3](https://github.com/HighCWu/control-lora-v3) by HighCWu
|
| - Dataset: [grayscale_image_aesthetic_3M](https://huggingface.co/datasets/latentcat/grayscale_image_aesthetic_3M) by latentcat
|
| - Training infrastructure: NVIDIA H100 80GB
|
| - LoRA implementation: [PEFT](https://github.com/huggingface/peft) by Hugging Face
|
|
|
| ## License
|
|
|
| Apache 2.0 License. The base SDXL model has separate license terms at [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0).
|
|
|
