--- license: apache-2.0 base_model: stabilityai/stable-diffusion-xl-base-1.0 tags: - stable-diffusion-xl - sdxl - text-to-image - diffusers - lora - control - controlnet - control-lora - brightness - grayscale - template:sd-lora widget: - text: "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" output: url: "https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/example.png" inference: true --- # ControlNet LoRA SDXL - Brightness Control (10k @ 1024×1024) A Control LoRA model trained on Stable Diffusion XL to control image generation through brightness/grayscale information. This model uses **LoRA (Low-Rank Adaptation)** combined with ControlNet architecture for efficient control, providing an **ultra-lightweight alternative** to full ControlNet with excellent pattern preservation. ## Model Description This Control LoRA enables brightness-based conditioning for SDXL image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts. ### Key Features: - 🎨 **Excellent brightness and pattern control** across multiple scales (0.5-2.0) - 🚀 **196x smaller than full ControlNet**: ~24MB vs ~4.7GB - ⚡ **Ultra-fast loading**: LoRA weights load in <1 second - 💡 **Flexible scale control**: Adjustable conditioning scale from 0.5 to 2.0+ - 🔄 **Compatible with ControlLoRA v3**: Uses the efficient ControlLoRA v3 architecture - 📦 **Minimal storage**: All checkpoints + final model = ~120MB total - 🖼️ **Native SDXL resolution**: Trained at 1024×1024 ### Intended Uses: - **Artistic QR code generation** (scale 1.0-1.5 recommended) - Image recoloring and colorization - Lighting control in text-to-image generation - Brightness-based pattern integration - Watermark and subtle pattern embedding - Photo enhancement and stylization ## Training Details ### Training Data Trained on 10,000 samples from `latentcat/grayscale_image_aesthetic_3M`: - High-quality aesthetic images - Paired with grayscale/brightness versions - Native resolution: 1024×1024 (SDXL native) ### Training Configuration | Parameter | Value | |-----------|-------| | **Base Model** | `stabilityai/stable-diffusion-xl-base-1.0` | | **Architecture** | ControlLoRA v3 (~7M trainable parameters) | | **LoRA Rank** | 16 | | **Extra Conv Rank** | 64 (conv_in layer) | | **Training Resolution** | 1024×1024 | | **Training Steps** | 313 (1 epoch) | | **Batch Size** | 8 per device | | **Gradient Accumulation** | 4 (effective batch: 32) | | **Learning Rate** | 1e-4 | | **Empty Prompts** | 10% (ControlLoRA v3 recommendation) | | **Mixed Precision** | BF16 | | **Hardware** | NVIDIA H100 80GB | | **Training Time** | ~21 minutes | | **Final Loss** | ~0.10-0.12 | ### Model Size Comparison | Model | Parameters | Size | Training | Resolution | |-------|-----------|------|----------|-----------| | **This Control LoRA** | ~7M | **~24MB** | 10k @ 1024 | 1024×1024 | | ControlNet (SDXL) | ~700M | 4.7GB | 100k @ 512 | 512×512 | | T2I Adapter (SDXL) | ~77M | 302MB | 100k @ 1024 | 1024×1024 | | Flux Control LoRA | ~7M | 25MB | 10k @ 512 | 512×512 | ## Usage ### Installation ```bash pip install diffusers transformers accelerate torch peft # Install ControlLoRA v3 git clone https://github.com/HighCWu/control-lora-v3 ``` ### Basic Usage ```python import torch import sys sys.path.insert(0, '/path/to/control-lora-v3') from pipeline_sdxl import StableDiffusionXLControlLoraV3Pipeline from model import UNet2DConditionModelEx from PIL import Image # Load UNet with LoRA support unet = UNet2DConditionModelEx.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", subfolder="unet", torch_dtype=torch.bfloat16, ) unet = unet.add_extra_conditions(["brightness"]) # Load SDXL Control LoRA pipeline pipe = StableDiffusionXLControlLoraV3Pipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", unet=unet, torch_dtype=torch.bfloat16, ) # Load Control LoRA weights pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness") pipe.to("cuda") # Load grayscale/brightness control image control_image = Image.open("path/to/grayscale_image.png") control_image = control_image.resize((1024, 1024)) # Generate image prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" image = pipe( prompt=prompt, image=control_image, num_inference_steps=30, guidance_scale=7.5, extra_condition_scale=1.0, # Controls conditioning strength height=1024, width=1024, ).images[0] image.save("output.png") ``` ### Adjusting Control Strength The `extra_condition_scale` parameter controls how strongly the brightness map influences generation: ```python # Subtle control (scale 0.5-0.7) image = pipe( prompt=prompt, image=control_image, extra_condition_scale=0.5, ... ).images[0] # Balanced control (scale 1.0-1.5) - Recommended for artistic QR codes image = pipe( prompt=prompt, image=control_image, extra_condition_scale=1.0, ... ).images[0] # Strong control (scale 1.5-2.0) image = pipe( prompt=prompt, image=control_image, extra_condition_scale=1.5, ... ).images[0] ``` ### Artistic QR Code Generation ```python import qrcode from PIL import Image # Generate QR code qr = qrcode.QRCode( version=1, error_correction=qrcode.constants.ERROR_CORRECT_H, box_size=10, border=4 ) qr.add_data("https://your-url.com") qr.make(fit=True) qr_image = qr.make_image(fill_color="black", back_color="white") qr_image = qr_image.resize((1024, 1024), Image.LANCZOS).convert("RGB") # Generate artistic QR code (scale 1.0-1.5 works best) image = pipe( prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography", image=qr_image, num_inference_steps=30, guidance_scale=7.5, extra_condition_scale=1.0, height=1024, width=1024, ).images[0] image.save("artistic_qr.png") ``` ### Using Different Checkpoints The model includes intermediate checkpoints from throughout training: ```python # Early checkpoint (25% - 2,500 samples) pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness", subfolder="checkpoint-78") # Mid checkpoint (50% - 5,000 samples) pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness", subfolder="checkpoint-156") # Late checkpoint (75% - 7,500 samples) pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness", subfolder="checkpoint-234") # Near-final checkpoint (99% - 9,984 samples) pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness", subfolder="checkpoint-312") # Final model (10,000 samples, main branch - recommended) pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl-10k", adapter_name="brightness") ``` ## Conditioning Scale Guide The `extra_condition_scale` parameter controls how strongly the brightness map influences generation: ### Recommended Scale Ranges | Scale | Behavior | Best For | |-------|----------|----------| | **0.5-0.7** | Subtle artistic integration with hints of pattern | Natural images, soft lighting hints | | **0.7-1.0** | Light control - visible structure with artistic freedom | Artistic images, creative reinterpretation | | **1.0-1.5** | **🔥 Balanced control** | **Artistic QR codes, watermarks** (recommended) | | **1.5-2.0** | Strong control - clear patterns with artistic overlay | Geometric patterns, structured designs | | **2.0+** | Maximum control - dominant patterns | Strong brightness maps, technical applications | ## Performance Comparison ### vs Full ControlNet (SDXL) | Metric | ControlNet (SDXL) | This Control LoRA | Advantage | |--------|-------------------|-------------------|-----------| | **Parameters** | ~700M | ~7M | **100x smaller** | | **Model Size** | 4.7GB | 24MB | **196x smaller** | | **Load Time** | ~5-10 seconds | **<1 second** | **10x faster loading** | | **Storage (w/ checkpoints)** | ~18.8GB | **~120MB** | **157x less storage** | | **Training Time** | ~3 hours | **21 minutes** | **8.5x faster** | | **Pattern Preservation @ Scale 1.0** | Excellent | **Excellent** | Comparable quality | | **Flexibility** | Fixed architecture | **Adjustable weights** | More versatile | ### vs T2I Adapter (SDXL) | Metric | T2I Adapter (SDXL) | This Control LoRA | Advantage | |--------|-------------------|-------------------|-----------| | **Parameters** | ~77M | ~7M | **11x smaller** | | **Model Size** | 302MB | 24MB | **12.6x smaller** | | **Training Samples** | 100k | 10k | **More efficient** | | **Architecture** | Separate adapter | **Integrated LoRA** | Simpler loading | ## Checkpoint Progression Analysis The model includes checkpoints from throughout training: 1. **checkpoint-78**: 25% complete (2,500 samples) 2. **checkpoint-156**: 50% complete (5,000 samples) 3. **checkpoint-234**: 75% complete (7,500 samples) 4. **checkpoint-312**: 99% complete (9,984 samples) 5. **Final model**: 100% complete (10,000 samples - main branch) ### Visual Comparison Each comparison shows QR input + all 7 conditioning scales (0.25, 0.5, 0.7, 0.75, 1.0, 1.25, 1.5) for a specific checkpoint: #### Checkpoint 78 (25% trained, 2,500 samples) ![Checkpoint-78 Scale Progression](https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/controlnet_lora_scale_comparison_checkpoint-78.png) #### Checkpoint 156 (50% trained, 5,000 samples) ![Checkpoint-156 Scale Progression](https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/controlnet_lora_scale_comparison_checkpoint-156.png) #### Checkpoint 234 (75% trained, 7,500 samples) ![Checkpoint-234 Scale Progression](https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/controlnet_lora_scale_comparison_checkpoint-234.png) #### Checkpoint 312 (99% trained, 9,984 samples) ![Checkpoint-312 Scale Progression](https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/controlnet_lora_scale_comparison_checkpoint-312.png) #### Final Model (100% trained, 10,000 samples) - Recommended ![Final Model Scale Progression](https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/controlnet_lora_scale_comparison_final.png) ### Key Observations All checkpoints show consistent, high-quality performance across scales. The progression analysis reveals: 1. **Early Checkpoint (78 steps, 2.5k samples)**: - Good pattern awareness, developing control - More artistic interpretation of prompts - Recommended scales: 0.7-1.2 2. **Mid Checkpoints (156-234 steps, 5k-7.5k samples)**: - Strong balance between control and creativity - Stable pattern preservation - Recommended scales: 0.8-1.5 3. **Final Model (313 steps, 10k samples)**: - Maximum control capability - Excellent pattern preservation at all scales - Recommended scales: 0.7-2.0 ### No Overfitting Observed Unlike larger models trained on 100k samples, this 10k Control LoRA shows **no signs of overfitting**: - Consistent improvement throughout training - Final checkpoint is recommended for production use - 10k samples appears optimal for LoRA-based control training ## When to Use This Model ### ✅ Use This Control LoRA When: - Creating **artistic QR codes** with SDXL quality (scale 1.0-1.5) - Need **minimal storage overhead** (<30MB per checkpoint) - Want **fast model loading** (<1 second) - Building **production applications** requiring small model sizes - Working with **SDXL** as base model - Require **flexible control strength** via extra_condition_scale - Need **multiple checkpoints** without massive storage (120MB total vs 18.8GB) - Training on **limited data** (10k samples sufficient) ### ⚠️ Consider Alternatives When: - Need **full ControlNet** features with extremely precise control - Working with existing **T2I Adapter** pipelines - Require **different control types** (pose, depth, etc.) - train separate LoRAs ## Limitations ### Current Limitations - **ControlLoRA v3 dependency**: Requires custom pipeline code (not in main diffusers yet) - **Grayscale conditioning only**: Trained specifically for brightness/grayscale control - **Single control type**: Only brightness, not other conditioning types - **Custom code required**: Need to include ControlLoRA v3 files ### Recommendations - For **SDXL generation**, use this Control LoRA - For **multiple control types**, train separate LoRAs and combine - Experiment with **scales 1.0-1.5** for most use cases - Use **final model** for best results ## Training Script ```bash accelerate launch --mixed_precision="bf16" train_sdxl.py \ --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \ --dataset_name="" \ --conditioning_image_column="conditioning_image" \ --image_column="image" \ --caption_column="text" \ --output_dir="./controlnet-lora-brightness-sdxl-10k" \ --mixed_precision="bf16" \ --resolution=1024 \ --learning_rate=1e-4 \ --proportion_empty_prompts=0.1 \ --rank=16 \ --lora_adapter_name="brightness" \ --extra_lora_rank_modules conv_in \ --extra_lora_ranks 64 \ --half_or_full_lora=half_skip_attn \ --train_batch_size=8 \ --num_train_epochs=1 \ --gradient_accumulation_steps=4 \ --gradient_checkpointing \ --checkpointing_steps=78 \ --validation_steps=78 \ --validation_image="validation_qr.png" \ --validation_prompt="a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" \ --num_validation_images=4 \ --seed=42 \ --dataloader_num_workers=4 \ --tracker_project_name="controlnet-lora-brightness-sdxl-10k" \ --report_to="wandb" \ --enable_xformers_memory_efficient_attention \ --use_8bit_adam \ --init_lora_weights="gaussian" ``` ## Available Checkpoints All checkpoints are available in the main branch: - **Root directory**: Final model (10,000 samples, recommended) - **checkpoint-78/**: Early checkpoint (2,500 samples, 25% trained) - **checkpoint-156/**: Mid checkpoint (5,000 samples, 50% trained) - **checkpoint-234/**: Late checkpoint (7,500 samples, 75% trained) - **checkpoint-312/**: Near-final checkpoint (9,984 samples, 99% trained) ## Citation ```bibtex @misc{controlnet-lora-brightness-sdxl, author = {Oysiyl}, title = {ControlNet LoRA SDXL - Brightness Control (10k @ 1024×1024)}, year = {2026}, publisher = {HuggingFace}, journal = {HuggingFace Model Hub}, howpublished = {\url{https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl}} } ``` ## Acknowledgments - Built with [🤗 Diffusers](https://github.com/huggingface/diffusers) - Base model: [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) by Stability AI - ControlLoRA v3: [control-lora-v3](https://github.com/HighCWu/control-lora-v3) by HighCWu - Dataset: [grayscale_image_aesthetic_3M](https://huggingface.co/datasets/latentcat/grayscale_image_aesthetic_3M) by latentcat - Training infrastructure: NVIDIA H100 80GB - LoRA implementation: [PEFT](https://github.com/huggingface/peft) by Hugging Face ## License Apache 2.0 License. The base SDXL model has separate license terms at [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0).