---
license: apache-2.0
base_model: black-forest-labs/FLUX.1-dev
tags:
- flux
- flux-diffusers
- text-to-image
- diffusers
- lora
- control
- controlnet
- brightness
- grayscale
- template:sd-lora
widget:
- text: "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"
  output:
    url: "https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k/resolve/main/example.png"
inference: true
---

# FLUX ControlNet LoRA - Brightness Control (10k @ 1024×1024)

A Control LoRA model trained on FLUX.1-dev to control image generation through brightness/grayscale information. This model uses **LoRA (Low-Rank Adaptation)** combined with ControlNet architecture for efficient control, providing an **ultra-lightweight control mechanism** for FLUX's powerful 12B parameter model.

## Model Description

This Control LoRA enables brightness-based conditioning for FLUX image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.

### Key Features:

- 🎨 **Excellent brightness and pattern control** with FLUX's superior quality
- 🚀 **Compact model size**: ~685MB per checkpoint
- ⚡ **Fast inference**: Leverages FLUX's optimized architecture
- 💡 **Fixed conditioning strength**: Baked into LoRA weights during training
- 🔄 **Compatible with Diffusers**: Uses standard FluxControlPipeline
- 📦 **Multiple checkpoints**: Track training progression at 25%, 50%, 75%, 100%
- 🖼️ **Native FLUX resolution**: Trained at 1024×1024
- 🎯 **XLabs-proven parameters**: Validated hyperparameters for stability

### Intended Uses:

- **Artistic QR code generation**
- Image recoloring and colorization
- Lighting control in text-to-image generation
- Brightness-based pattern integration
- Watermark and subtle pattern embedding
- Photo enhancement and stylization

## Training Details

### Training Data

Trained on 10,000 samples from `latentcat/grayscale_image_aesthetic_3M`:
- High-quality aesthetic images
- Paired with grayscale/brightness versions
- Native resolution: 1024×1024

### Training Configuration

| Parameter | Value |
|-----------|-------|
| **Base Model** | `black-forest-labs/FLUX.1-dev` |
| **Model Size** | 12B parameters |
| **Architecture** | FLUX Control LoRA |
| **LoRA Rank** | 128 |
| **Training Resolution** | 1024×1024 |
| **Training Steps** | 1,250 (1 epoch) |
| **Batch Size** | 1 per device |
| **Gradient Accumulation** | 8 (effective batch: 8) |
| **Learning Rate** | 2e-5 constant (XLabs proven) |
| **LR Warmup** | 10 steps (XLabs proven) |
| **Weight Decay** | 0.01 (XLabs proven) |
| **Max Grad Norm** | 1.0 (XLabs proven) |
| **Empty Prompts** | 20% |
| **Init Method** | Default |
| **Mixed Precision** | BF16 |
| **Hardware** | NVIDIA H100 80GB |
| **Training Time** | ~3h 14min |
| **Memory Usage** | ~45GB VRAM |
| **Final Loss** | ~0.08-0.12 |

### XLabs Proven Parameters

This model uses hyperparameters validated by [XLabs-AI](https://github.com/XLabs-AI/x-flux) for FLUX ControlNet training:

- **Learning Rate 2e-5**: 5x lower than typical SDXL rates, critical for FLUX's 12B parameters
- **Warmup 10 steps**: Gradual LR increase for training stability
- **Weight Decay 0.01**: L2 regularization to prevent overfitting
- **Max Grad Norm 1.0**: Gradient clipping to prevent explosion

These parameters provide stable training and prevent divergence in large models.

### Model Size Comparison

| Model | Parameters | Size | Training | Resolution |
|-------|-----------|------|----------|-----------|
| **This Control LoRA** | ~12B | **~685MB** | 10k @ 1024 | 1024×1024 |
| ControlNet (SDXL) | ~700M | 4.7GB | 100k @ 512 | 512×512 |
| T2I Adapter (SDXL) | ~77M | 302MB | 100k @ 1024 | 1024×1024 |
| SDXL Control LoRA | ~7M | 24MB | 100k @ 1024 | 1024×1024 |

## Usage

### Installation

```bash
pip install diffusers transformers accelerate torch
```

### Basic Usage

```python
import torch
import sys
sys.path.insert(0, '/path/to/diffusers/src')

from diffusers import FluxControlPipeline
from PIL import Image

# Load FLUX Control Pipeline
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((1024, 1024))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,  # FLUX default
    guidance_scale=3.5,       # FLUX default
    height=1024,
    width=1024,
).images[0]

image.save("output.png")
```

### Adjusting Control Strength

You can control the LoRA conditioning strength using `set_adapters()` with `adapter_weights`:

```python
# Load pipeline and LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Set adapter weight (scale) - controls conditioning strength
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Now generate with adjusted strength
image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=1024,
    width=1024,
).images[0]
```

**Recommended Scale Ranges:**
- **0.5-0.7**: Subtle control with hints of pattern
- **0.75-1.0**: Balanced control (recommended for artistic QR codes)
- **1.0-1.25**: Strong control with clear patterns
- **1.25-1.5**: Very strong control

**Note:** Unlike SDXL ControlNet which uses `controlnet_conditioning_scale`, FLUX Control LoRA uses `set_adapters()` with `adapter_weights` to control conditioning strength.

### Artistic QR Code Generation

```python
import qrcode
from PIL import Image
import torch
from diffusers import FluxControlPipeline

# Generate QR code
qr = qrcode.QRCode(
    version=None,
    error_correction=qrcode.constants.ERROR_CORRECT_H,  # HIGH (30% tolerance)
    box_size=16,   # 16 pixels per module
    border=6,      # 6 modules border
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((1024, 1024), Image.LANCZOS).convert("RGB")

# Load pipeline with Control LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Generate artistic QR code
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    control_image=qr_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=1024,
    width=1024,
).images[0]

image.save("artistic_qr.png")
```

### Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

```python
# Early checkpoint (25% - 2,500 samples, step 312)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-312")

# Mid checkpoint (50% - 5,000 samples, step 624)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-624")

# Late checkpoint (75% - 7,500 samples, step 936)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-936")

# Final model (10,000 samples, main branch - recommended)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
```

## Scale Comparison Grids

Visual comparison of different adapter weights (scales) for each checkpoint. Each grid shows the same prompt and QR code at scales 0.5, 0.75, 1.0, 1.25, and 1.5.

### Checkpoint-312 (25% trained, 2,500 samples)

![Checkpoint-312 Scale Comparison](examples/flux_controlnet_lora_scale_comparison_checkpoint-312.png)

**Scale progression:** From subtle (0.5) to very strong (1.5) control. Early checkpoint shows dense pattern integration with visible QR structure across all scales.

### Checkpoint-624 (50% trained, 5,000 samples)
![Checkpoint-624 Scale Comparison](examples/flux_controlnet_lora_scale_comparison_checkpoint-624.png)

**Scale progression:** Improved artistic balance at mid-training. Natural garden scenes emerge with adjustable pattern strength.

### Checkpoint-936 (75% trained, 7,500 samples)
![Checkpoint-936 Scale Comparison](examples/flux_controlnet_lora_scale_comparison_checkpoint-936.png)

**Scale progression:** Refined artistic quality. Excellent detail preservation with smooth scale transitions.

### Final Model (100% trained, 10,000 samples)
![Final Checkpoint Scale Comparison](examples/flux_controlnet_lora_scale_comparison_final.png)

**Scale progression:** Best overall quality. Recommended scales: 0.75-1.0 for artistic QR codes, 1.0-1.25 for pattern integration.

### Scale Selection Guide

Based on the comparison grids:

| Scale | Control Strength | Best For | Visual Effect |
|-------|-----------------|----------|---------------|
| **0.5** | Very subtle | Natural images, minimal pattern hints | Barely visible patterns, maximum artistic freedom |
| **0.75** | Light | Artistic images with soft pattern integration | Balanced artistic quality with gentle patterns |
| **1.0** | Balanced | **Artistic QR codes** (recommended) | Clear patterns with strong artistic elements |
| **1.25** | Strong | Pattern-guided art, structured designs | Dominant patterns with artistic overlay |
| **1.5** | Very strong | Technical applications, geometric patterns | Maximum pattern visibility, reduced artistic freedom |

## Checkpoint Progression

All checkpoints are included on the main branch to show training progression:

### Checkpoint-312 (25% trained, 2,500 samples)
<img src="examples/flux_controlnet_lora_checkpoint-312.png" width="512" alt="Checkpoint 312">

Early training stage - dense pattern integration with strong QR visibility.

### Checkpoint-624 (50% trained, 5,000 samples)
<img src="examples/flux_controlnet_lora_checkpoint-624.png" width="512" alt="Checkpoint 624">

Mid-training - improved artistic balance with natural perspective and composition.

### Checkpoint-936 (75% trained, 7,500 samples)
<img src="examples/flux_controlnet_lora_checkpoint-936.png" width="512" alt="Checkpoint 936">

Advanced training - refined floral composition with excellent detail and artistic quality.

### Final Model (100% trained, 10,000 samples)
<img src="examples/flux_controlnet_lora_final.png" width="512" alt="Final Model">

Final model - unique artistic interpretation with frame effect and natural scene integration.

### Original QR Code
<img src="examples/flux_controlnet_lora_qr_original.png" width="256" alt="QR Original">

Validation QR code used during training (https://google.com, HIGH error correction).

## Performance Comparison

### vs SDXL Control LoRA

| Metric | SDXL Control LoRA | This FLUX Control LoRA | Notes |
|--------|-------------------|------------------------|-------|
| **Base Model** | SDXL (2.6B) | FLUX.1-dev (12B) | **4.6x larger model** |
| **Parameters** | ~7M | ~12B | Full model parameters |
| **Model Size** | 24MB | 685MB | Includes full weights |
| **LoRA Rank** | 16 | 128 | **8x higher rank** |
| **Training Samples** | 100k | 10k | Fewer samples |
| **Training Time** | ~21 min | ~3h 14min | Larger model overhead |
| **Variable Scale** | ✅ Yes (extra_condition_scale) | ✅ Yes (set_adapters) | Different methods |
| **Quality** | Excellent | **Superior** | FLUX advantage |

### vs Full ControlNet

| Metric | ControlNet (SDXL) | This FLUX Control LoRA | Advantage |
|--------|-------------------|------------------------|-----------|
| **Model Size** | 4.7GB | 685MB | **7x smaller** |
| **Storage (w/ checkpoints)** | ~18.8GB | **~2.7GB** | **7x less storage** |
| **Training Time** | ~3 hours | 3h 14min | Comparable |
| **Flexibility** | Variable scale | Fixed strength | Trade-off |

## When to Use This Model

### ✅ Use This FLUX Control LoRA When:

- Creating **artistic images** with FLUX's superior quality
- Need **brightness-based control** for FLUX generation
- Want **production-ready artistic QR codes**
- Working with **FLUX.1-dev** as base model
- Require **checkpoint progression** for analysis
- Need **efficient storage** vs full ControlNet

### ⚠️ Consider Alternatives When:

- Working with **smaller datasets** (SDXL trains faster with 10k samples)
- Need **minimal model size** (SDXL Control LoRA is 24MB vs 685MB)
- Require **extremely precise control** (use full ControlNet)
- Need **multiple control types** - train separate LoRAs

## Limitations

### Current Limitations

- **FLUX-specific**: Only works with FLUX.1-dev base model
- **Grayscale conditioning only**: Trained specifically for brightness/grayscale control
- **Single control type**: Only brightness, not other conditioning types (pose, depth, etc.)
- **Larger model size**: 685MB vs SDXL's 24MB (includes more parameters)
- **Requires diffusers**: Uses standard FluxControlPipeline from diffusers library

### Recommendations

- Use **set_adapters()** with **adapter_weights=[0.75-1.0]** for best artistic QR results
- For **multiple control types**, train separate LoRAs
- Use **final model** for best results (recommended)
- Use **checkpoint-936 (75%)** for alternative artistic style
- Experiment with different scales (0.5-1.5) to find the right balance

## Training Script

```bash
#!/bin/bash
set -e

# Configuration
LOCAL_DATASET_10K="$HOME/.cache/huggingface/datasets_partial/grayscale_10k/combined"
OUTPUT_DIR="$HOME/controlnet-lora-brightness-flux-10k"
VALIDATION_QR="$HOME/validation_qr.png"
CHECKPOINT_STEPS=78  # Checkpoint every 2,500 samples

# Change to training directory
cd diffusers/examples/flux-control

# Training with XLabs proven parameters
accelerate launch --mixed_precision="bf16" train_control_lora_flux.py \
  --pretrained_model_name_or_path="black-forest-labs/FLUX.1-dev" \
  --dataset_name="$LOCAL_DATASET_10K" \
  --output_dir="$OUTPUT_DIR" \
  --mixed_precision="bf16" \
  --resolution=1024 \
  --learning_rate=2e-5 \
  --lr_scheduler=constant \
  --lr_warmup_steps=10 \
  --adam_weight_decay=0.01 \
  --max_grad_norm=1.0 \
  --proportion_empty_prompts=0.2 \
  --rank=128 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=8 \
  --num_train_epochs=1 \
  --gradient_checkpointing \
  --checkpointing_steps=$CHECKPOINT_STEPS \
  --validation_steps=$CHECKPOINT_STEPS \
  --validation_image="$VALIDATION_QR" \
  --validation_prompt="a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" \
  --num_validation_images=4 \
  --guidance_scale=3.5 \
  --seed=42 \
  --dataloader_num_workers=4 \
  --report_to="wandb" \
  --tracker_project_name="controlnet-lora-brightness-flux-10k"
```

## Available Checkpoints

All checkpoints are available in the main branch:

- **Root directory**: Final model (10,000 samples, 1,250 steps, **recommended**)
- **checkpoint-312/**: Early checkpoint (2,500 samples, 25% trained)
- **checkpoint-624/**: Mid checkpoint (5,000 samples, 50% trained)
- **checkpoint-936/**: Late checkpoint (7,500 samples, 75% trained)

## Key Differences: FLUX vs SDXL

| Aspect | SDXL Control LoRA | FLUX Control LoRA |
|--------|-------------------|-------------------|
| **Model Size** | 2.6B params (UNet) | 12B params |
| **LoRA Rank** | 16 | 128 (8x higher) |
| **Model File Size** | ~24MB | ~685MB |
| **Learning Rate** | 1e-4 | 2e-5 (XLabs proven) |
| **Warmup Steps** | 0 | 10 (XLabs) |
| **Weight Decay** | Not used | 0.01 (XLabs) |
| **Max Grad Norm** | Not used | 1.0 (XLabs) |
| **Variable Scale** | ✅ Yes (extra_condition_scale) | ✅ Yes (set_adapters) |
| **Inference Steps** | 30 | 28 |
| **Guidance Scale** | 7.5 | 3.5 |
| **Training Time (10k)** | ~21 min | ~3h 14min |
| **Memory (1024)** | ~40GB | ~45GB |

## Troubleshooting

### OOM (Out of Memory) Errors

If you encounter OOM during inference:

```python
# Enable CPU offload
pipe.enable_model_cpu_offload()

# Or use sequential CPU offload (slower but less memory)
pipe.enable_sequential_cpu_offload()
```

### Loading Errors

Make sure you have the latest diffusers:

```bash
pip install --upgrade diffusers transformers accelerate
```

### Quality Issues

- Use **guidance_scale=3.5** (FLUX default)
- Use **num_inference_steps=28** or higher
- Try different checkpoints (final model recommended)
- Ensure control image is 1024×1024 resolution

## Citation

```bibtex
@misc{controlnet-lora-brightness-flux,
  author = {Oysiyl},
  title = {FLUX ControlNet LoRA - Brightness Control (10k @ 1024×1024)},
  year = {2026},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/Oysiyl/controlnet-lora-brightness-flux}}
}
```

## Acknowledgments

- Built with [🤗 Diffusers](https://github.com/huggingface/diffusers)
- Base model: [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) by Black Forest Labs
- XLabs proven parameters: [x-flux](https://github.com/XLabs-AI/x-flux) by XLabs-AI
- Dataset: [grayscale_image_aesthetic_3M](https://huggingface.co/datasets/latentcat/grayscale_image_aesthetic_3M) by latentcat
- Training infrastructure: NVIDIA H100 80GB
- LoRA implementation: [PEFT](https://github.com/huggingface/peft) by Hugging Face

## License

Apache 2.0 License. The base FLUX.1-dev model has separate license terms at [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev).