README.md

metadata

license: apache-2.0
base_model: stabilityai/stable-diffusion-xl-base-1.0
tags:
  - stable-diffusion-xl
  - sdxl
  - text-to-image
  - diffusers
  - lora
  - control
  - controlnet
  - control-lora
  - brightness
  - grayscale
  - template:sd-lora
widget:
  - text: >-
      a beautiful garden scene with colorful flowers and butterflies, highly
      detailed, professional photography, vibrant colors
    output:
      url: >-
        https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl/resolve/main/examples/example.png
inference: true

ControlNet LoRA SDXL - Brightness Control (10k @ 1024×1024)

A Control LoRA model trained on Stable Diffusion XL to control image generation through brightness/grayscale information. This model uses LoRA (Low-Rank Adaptation) combined with ControlNet architecture for efficient control, providing an ultra-lightweight alternative to full ControlNet with excellent pattern preservation.

Model Description

This Control LoRA enables brightness-based conditioning for SDXL image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.

Key Features:

• 🎨 Excellent brightness and pattern control across multiple scales (0.5-2.0)
• 🚀 196x smaller than full ControlNet: ~24MB vs ~4.7GB
• ⚡ Ultra-fast loading: LoRA weights load in <1 second
• 💡 Flexible scale control: Adjustable conditioning scale from 0.5 to 2.0+
• 🔄 Compatible with ControlLoRA v3: Uses the efficient ControlLoRA v3 architecture
• 📦 Minimal storage: All checkpoints + final model = ~120MB total
• 🖼️ Native SDXL resolution: Trained at 1024×1024

Intended Uses:

• Artistic QR code generation (scale 1.0-1.5 recommended)
• Image recoloring and colorization
• Lighting control in text-to-image generation
• Brightness-based pattern integration
• Watermark and subtle pattern embedding
• Photo enhancement and stylization

Training Details

Training Data

Trained on 10,000 samples from latentcat/grayscale_image_aesthetic_3M:

• High-quality aesthetic images
• Paired with grayscale/brightness versions
• Native resolution: 1024×1024 (SDXL native)

Training Configuration

Parameter	Value
Base Model	stabilityai/stable-diffusion-xl-base-1.0
Architecture	ControlLoRA v3 (~7M trainable parameters)
LoRA Rank	16
Extra Conv Rank	64 (conv_in layer)
Training Resolution	1024×1024
Training Steps	313 (1 epoch)
Batch Size	8 per device
Gradient Accumulation	4 (effective batch: 32)
Learning Rate	1e-4
Empty Prompts	10% (ControlLoRA v3 recommendation)
Mixed Precision	BF16
Hardware	NVIDIA H100 80GB
Training Time	~21 minutes
Final Loss	~0.10-0.12

Model Size Comparison

Model	Parameters	Size	Training	Resolution
This Control LoRA	~7M	~24MB	10k @ 1024	1024×1024
ControlNet (SDXL)	~700M	4.7GB	100k @ 512	512×512
T2I Adapter (SDXL)	~77M	302MB	100k @ 1024	1024×1024
Flux Control LoRA	~7M	25MB	10k @ 512	512×512

Usage

Installation

pip install diffusers transformers accelerate torch peft
# Install ControlLoRA v3
git clone https://github.com/HighCWu/control-lora-v3

Basic Usage

import torch
import sys
sys.path.insert(0, '/path/to/control-lora-v3')

from pipeline_sdxl import StableDiffusionXLControlLoraV3Pipeline
from model import UNet2DConditionModelEx
from PIL import Image

# Load UNet with LoRA support
unet = UNet2DConditionModelEx.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    subfolder="unet",
    torch_dtype=torch.bfloat16,
)
unet = unet.add_extra_conditions(["brightness"])

# Load SDXL Control LoRA pipeline
pipe = StableDiffusionXLControlLoraV3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    unet=unet,
    torch_dtype=torch.bfloat16,
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness")
pipe.to("cuda")

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((1024, 1024))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    image=control_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    extra_condition_scale=1.0,  # Controls conditioning strength
    height=1024,
    width=1024,
).images[0]

image.save("output.png")

Adjusting Control Strength

The extra_condition_scale parameter controls how strongly the brightness map influences generation:

# Subtle control (scale 0.5-0.7)
image = pipe(
    prompt=prompt,
    image=control_image,
    extra_condition_scale=0.5,
    ...
).images[0]

# Balanced control (scale 1.0-1.5) - Recommended for artistic QR codes
image = pipe(
    prompt=prompt,
    image=control_image,
    extra_condition_scale=1.0,
    ...
).images[0]

# Strong control (scale 1.5-2.0)
image = pipe(
    prompt=prompt,
    image=control_image,
    extra_condition_scale=1.5,
    ...
).images[0]

Artistic QR Code Generation

import qrcode
from PIL import Image

# Generate QR code
qr = qrcode.QRCode(
    version=1,
    error_correction=qrcode.constants.ERROR_CORRECT_H,
    box_size=10,
    border=4
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((1024, 1024), Image.LANCZOS).convert("RGB")

# Generate artistic QR code (scale 1.0-1.5 works best)
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    image=qr_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    extra_condition_scale=1.0,
    height=1024,
    width=1024,
).images[0]

image.save("artistic_qr.png")

Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

# Early checkpoint (25% - 2,500 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-78")

# Mid checkpoint (50% - 5,000 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-156")

# Late checkpoint (75% - 7,500 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-234")

# Near-final checkpoint (99% - 9,984 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-312")

# Final model (10,000 samples, main branch - recommended)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl-10k",
                       adapter_name="brightness")

Conditioning Scale Guide

The extra_condition_scale parameter controls how strongly the brightness map influences generation:

Recommended Scale Ranges

Scale	Behavior	Best For
0.5-0.7	Subtle artistic integration with hints of pattern	Natural images, soft lighting hints
0.7-1.0	Light control - visible structure with artistic freedom	Artistic images, creative reinterpretation
1.0-1.5	🔥 Balanced control	Artistic QR codes, watermarks (recommended)
1.5-2.0	Strong control - clear patterns with artistic overlay	Geometric patterns, structured designs
2.0+	Maximum control - dominant patterns	Strong brightness maps, technical applications

Performance Comparison

vs Full ControlNet (SDXL)

Metric	ControlNet (SDXL)	This Control LoRA	Advantage
Parameters	~700M	~7M	100x smaller
Model Size	4.7GB	24MB	196x smaller
Load Time	~5-10 seconds	<1 second	10x faster loading
Storage (w/ checkpoints)	~18.8GB	~120MB	157x less storage
Training Time	~3 hours	21 minutes	8.5x faster
Pattern Preservation @ Scale 1.0	Excellent	Excellent	Comparable quality
Flexibility	Fixed architecture	Adjustable weights	More versatile

vs T2I Adapter (SDXL)

Metric	T2I Adapter (SDXL)	This Control LoRA	Advantage
Parameters	~77M	~7M	11x smaller
Model Size	302MB	24MB	12.6x smaller
Training Samples	100k	10k	More efficient
Architecture	Separate adapter	Integrated LoRA	Simpler loading

Checkpoint Progression Analysis

The model includes checkpoints from throughout training:

1. checkpoint-78: 25% complete (2,500 samples)
2. checkpoint-156: 50% complete (5,000 samples)
3. checkpoint-234: 75% complete (7,500 samples)
4. checkpoint-312: 99% complete (9,984 samples)
5. Final model: 100% complete (10,000 samples - main branch)

Visual Comparison

Each comparison shows QR input + all 7 conditioning scales (0.25, 0.5, 0.7, 0.75, 1.0, 1.25, 1.5) for a specific checkpoint:

Checkpoint 78 (25% trained, 2,500 samples)

Checkpoint 156 (50% trained, 5,000 samples)

Checkpoint 234 (75% trained, 7,500 samples)

Checkpoint 312 (99% trained, 9,984 samples)

Final Model (100% trained, 10,000 samples) - Recommended

Key Observations

All checkpoints show consistent, high-quality performance across scales. The progression analysis reveals:

1. Early Checkpoint (78 steps, 2.5k samples):

   • Good pattern awareness, developing control
   • More artistic interpretation of prompts
   • Recommended scales: 0.7-1.2

2. Mid Checkpoints (156-234 steps, 5k-7.5k samples):

   • Strong balance between control and creativity
   • Stable pattern preservation
   • Recommended scales: 0.8-1.5

3. Final Model (313 steps, 10k samples):

   • Maximum control capability
   • Excellent pattern preservation at all scales
   • Recommended scales: 0.7-2.0

No Overfitting Observed

Unlike larger models trained on 100k samples, this 10k Control LoRA shows no signs of overfitting:

• Consistent improvement throughout training
• Final checkpoint is recommended for production use
• 10k samples appears optimal for LoRA-based control training

When to Use This Model

✅ Use This Control LoRA When:

• Creating artistic QR codes with SDXL quality (scale 1.0-1.5)
• Need minimal storage overhead (<30MB per checkpoint)
• Want fast model loading (<1 second)
• Building production applications requiring small model sizes
• Working with SDXL as base model
• Require flexible control strength via extra_condition_scale
• Need multiple checkpoints without massive storage (120MB total vs 18.8GB)
• Training on limited data (10k samples sufficient)

⚠️ Consider Alternatives When:

• Need full ControlNet features with extremely precise control
• Working with existing T2I Adapter pipelines
• Require different control types (pose, depth, etc.) - train separate LoRAs

Limitations

Current Limitations

• ControlLoRA v3 dependency: Requires custom pipeline code (not in main diffusers yet)
• Grayscale conditioning only: Trained specifically for brightness/grayscale control
• Single control type: Only brightness, not other conditioning types
• Custom code required: Need to include ControlLoRA v3 files

Recommendations

• For SDXL generation, use this Control LoRA
• For multiple control types, train separate LoRAs and combine
• Experiment with scales 1.0-1.5 for most use cases
• Use final model for best results

Training Script

accelerate launch --mixed_precision="bf16" train_sdxl.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --dataset_name="<path_to_10k_dataset>" \
  --conditioning_image_column="conditioning_image" \
  --image_column="image" \
  --caption_column="text" \
  --output_dir="./controlnet-lora-brightness-sdxl-10k" \
  --mixed_precision="bf16" \
  --resolution=1024 \
  --learning_rate=1e-4 \
  --proportion_empty_prompts=0.1 \
  --rank=16 \
  --lora_adapter_name="brightness" \
  --extra_lora_rank_modules conv_in \
  --extra_lora_ranks 64 \
  --half_or_full_lora=half_skip_attn \
  --train_batch_size=8 \
  --num_train_epochs=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --checkpointing_steps=78 \
  --validation_steps=78 \
  --validation_image="validation_qr.png" \
  --validation_prompt="a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" \
  --num_validation_images=4 \
  --seed=42 \
  --dataloader_num_workers=4 \
  --tracker_project_name="controlnet-lora-brightness-sdxl-10k" \
  --report_to="wandb" \
  --enable_xformers_memory_efficient_attention \
  --use_8bit_adam \
  --init_lora_weights="gaussian"

Available Checkpoints

All checkpoints are available in the main branch:

• Root directory: Final model (10,000 samples, recommended)
• checkpoint-78/: Early checkpoint (2,500 samples, 25% trained)
• checkpoint-156/: Mid checkpoint (5,000 samples, 50% trained)
• checkpoint-234/: Late checkpoint (7,500 samples, 75% trained)
• checkpoint-312/: Near-final checkpoint (9,984 samples, 99% trained)

Citation

@misc{controlnet-lora-brightness-sdxl,
  author = {Oysiyl},
  title = {ControlNet LoRA SDXL - Brightness Control (10k @ 1024×1024)},
  year = {2026},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/Oysiyl/controlnet-lora-brightness-sdxl}}
}

Acknowledgments

• Built with 🤗 Diffusers
• Base model: Stable Diffusion XL by Stability AI
• ControlLoRA v3: control-lora-v3 by HighCWu
• Dataset: grayscale_image_aesthetic_3M by latentcat
• Training infrastructure: NVIDIA H100 80GB
• LoRA implementation: PEFT by Hugging Face

License

Apache 2.0 License. The base SDXL model has separate license terms at stabilityai/stable-diffusion-xl-base-1.0.