---
license: apache-2.0
base_model: black-forest-labs/FLUX.1-dev
tags:
- flux
- flux-diffusers
- text-to-image
- diffusers
- lora
- control
- controlnet
- brightness
- grayscale
- template:sd-lora
widget:
- text: "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"
  output:
    url: "https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k/resolve/main/example.png"
inference: true
---

# FLUX.1 Control LoRA - Brightness Control (10k @ 512×512)

A Control LoRA model trained on FLUX.1-dev to control image generation through brightness/grayscale information. This model uses **LoRA (Low-Rank Adaptation)** for efficient control, providing an **ultra-lightweight alternative** to ControlNet with excellent pattern preservation and minimal overhead.

## Model Description

This Control LoRA enables brightness-based conditioning for FLUX.1 image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.

### Key Features:

- 🎨 **Excellent brightness and pattern control** across all scales (0.25-1.5)
- 🚀 **188x smaller than SDXL ControlNet**: ~25MB vs ~4.7GB
- ⚡ **Ultra-fast loading**: LoRA weights load in <1 second
- 💡 **Flexible scale control**: Adjustable adapter weights from 0.25 to 1.5+
- 🔄 **Compatible with any FLUX.1-dev setup**: Just load the LoRA weights
- 📦 **Minimal storage**: All checkpoints + final model = ~125MB total

### Intended Uses:

- **Artistic QR code generation** (scale 0.7-1.0 recommended)
- Image recoloring and colorization
- Lighting control in text-to-image generation
- Brightness-based pattern integration
- Watermark and subtle pattern embedding
- Photo enhancement and stylization

## Training Details

### Training Data

Trained on 10,000 samples from `latentcat/grayscale_image_aesthetic_3M`:
- High-quality aesthetic images
- Paired with grayscale/brightness versions
- Resolution: 512×512 (balanced for FLUX.1)

### Training Configuration

| Parameter | Value |
|-----------|-------|
| **Base Model** | `black-forest-labs/FLUX.1-dev` |
| **Architecture** | Control LoRA (~7M trainable parameters) |
| **LoRA Rank** | 128 |
| **Training Resolution** | 512×512 |
| **Training Steps** | 2,500 (1 epoch) |
| **Batch Size** | 1 per device |
| **Gradient Accumulation** | 4 (effective batch: 4) |
| **Learning Rate** | 1e-4 |
| **Empty Prompts** | 50% (enables better control conditioning) |
| **Mixed Precision** | BF16 |
| **Hardware** | NVIDIA H100 80GB |
| **Training Time** | ~1.5 hours |
| **Final Loss** | 0.435 |

### Model Size Comparison

| Model | Parameters | Size | Training | Resolution |
|-------|-----------|------|----------|-----------|
| **This Control LoRA** | ~7M | **~25MB** | 10k @ 512 | 512×512 |
| ControlNet (SDXL) | ~700M | 4.7GB | 100k @ 512 | 512×512 |
| T2I Adapter (SDXL) | ~77M | 302MB | 100k @ 1024 | 1024×1024 |

## Usage

### Installation

```bash
pip install diffusers transformers accelerate torch sentencepiece peft
```

### Basic Usage

```python
from diffusers import FluxControlPipeline
import torch
from PIL import Image

# Load FLUX.1 Control pipeline
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")
pipe.to("cuda")

# Set adapter weight (scale) - controls conditioning strength
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((512, 512))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=512,
    width=512,
).images[0]

image.save("output.png")
```

### Adjusting Control Strength

The adapter weight controls how strongly the brightness map influences generation:

```python
# Subtle control (scale 0.25-0.5)
pipe.set_adapters(["default_0"], adapter_weights=[0.25])

# Balanced control (scale 0.7-1.0) - Recommended for artistic QR codes
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Strong control (scale 1.0-1.5)
pipe.set_adapters(["default_0"], adapter_weights=[1.25])
```

### Artistic QR Code Generation

```python
import qrcode
from PIL import Image

# Generate QR code
qr = qrcode.QRCode(
    version=1,
    error_correction=qrcode.constants.ERROR_CORRECT_H,
    box_size=10,
    border=4
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((512, 512), Image.LANCZOS).convert("RGB")

# Load pipeline and LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")
pipe.to("cuda")

# Set scale for good QR code readability (0.7-1.0 works best)
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Generate artistic QR code
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    control_image=qr_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=512,
    width=512,
).images[0]

image.save("artistic_qr.png")
```

### Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

```python
# Early checkpoint (25% - 2,500 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-625")

# Mid checkpoint (50% - 5,000 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-1250")

# Late checkpoint (75% - 7,500 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-1875")

# Final checkpoint (100% - 10,000 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-2500")

# Final model (same as checkpoint-2500, on main branch)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")
```

## Adapter Weight (Scale) Guide

The adapter weight controls how strongly the brightness map influences generation:

### Recommended Scale Ranges

| Scale | Behavior | Best For |
|-------|----------|----------|
| **0.25-0.5** | Subtle artistic integration with hints of pattern | Natural images, soft lighting hints |
| **0.5-0.7** | Light control - visible structure with strong artistic freedom | Artistic images, creative reinterpretation |
| **0.7-1.0** | **🔥 Balanced control** | **Artistic QR codes, watermarks** (recommended) |
| **1.0-1.25** | Strong control - clear patterns with artistic overlay | Geometric patterns, structured designs |
| **1.25-1.5** | Maximum control - dominant patterns | Strong brightness maps, technical applications |

### Visual Examples

#### Checkpoint Progression at Scale 0.75

This image shows how the model improves across checkpoints at scale 0.75 (recommended for QR codes):

![Checkpoint Progression at Scale 0.75](https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k/resolve/main/examples/scale_0.75_progression.png)

*From left to right: Original QR input, checkpoint-625 (25%), checkpoint-1250 (50%), checkpoint-1875 (75%), checkpoint-2500 (100%), final model*

#### Final Model - All Scales Comparison

This grid shows the final model's behavior across all scales (0.25 to 1.5):

![Final Model - All Scales](https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k/resolve/main/examples/final_model_all_scales.png)

*Grid showing scales: 0.25, 0.5, 0.7, 0.75, 1.0, 1.25, 1.5 - demonstrating the range from subtle hints to strong pattern preservation*

**Input QR Code:**

![QR Code Input](https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k/resolve/main/examples/qr_input.png)

## Performance Comparison

### vs ControlNet (SDXL)

| Metric | ControlNet (SDXL) | This Control LoRA | Advantage |
|--------|-------------------|-------------------|-----------|
| **Parameters** | ~700M | ~7M | **100x smaller** |
| **Model Size** | 4.7GB | 25MB | **188x smaller** |
| **Load Time** | ~5-10 seconds | **<1 second** | **10x faster loading** |
| **Storage (w/ checkpoints)** | ~18.8GB | **~125MB** | **150x less storage** |
| **Pattern Preservation @ Scale 1.0** | Excellent | **Excellent** | Comparable quality |
| **Flexibility** | Fixed architecture | **Adjustable weights** | More versatile |

### vs T2I Adapter (SDXL)

| Metric | T2I Adapter (SDXL) | This Control LoRA | Advantage |
|--------|-------------------|-------------------|-----------|
| **Parameters** | ~77M | ~7M | **11x smaller** |
| **Model Size** | 302MB | 25MB | **12x smaller** |
| **Base Model** | SDXL | **FLUX.1-dev** | More advanced base |
| **Training Samples** | 100k | 10k | **More efficient** |

## Checkpoint Progression Analysis

The model includes checkpoints from throughout training:

1. **checkpoint-625**: 25% complete (2,500 samples)
2. **checkpoint-1250**: 50% complete (5,000 samples)
3. **checkpoint-1875**: 75% complete (7,500 samples)
4. **checkpoint-2500**: 100% complete (10,000 samples)
5. **Final model**: Same as checkpoint-2500 (main branch)

### Key Observations

All checkpoints show consistent, high-quality performance across scales. The progression analysis reveals:

1. **Early Checkpoint (625 steps, 2.5k samples)**:
   - Good pattern awareness, developing control
   - More artistic interpretation of prompts
   - Recommended scales: 0.5-1.0

2. **Mid Checkpoints (1250-1875 steps, 5k-7.5k samples)**:
   - Strong balance between control and creativity
   - Stable pattern preservation
   - Recommended scales: 0.7-1.25

3. **Final Model (2500 steps, 10k samples)**:
   - Maximum control capability
   - Excellent pattern preservation at all scales
   - Recommended scales: 0.7-1.5

### No Overfitting Observed

Unlike larger models trained on 100k samples, this 10k Control LoRA shows **no signs of overfitting**:
- Consistent improvement throughout training
- Final checkpoint is recommended for production use
- 10k samples appears optimal for LoRA-based control training

## When to Use This Model

### ✅ Use This Control LoRA When:

- Creating **artistic QR codes** with FLUX.1 quality (scale 0.7-1.0)
- Need **minimal storage overhead** (<30MB per checkpoint)
- Want **fast model switching** between checkpoints (<1 second)
- Building **production applications** requiring small model sizes
- Working with **FLUX.1-dev** as base model
- Require **flexible control strength** via adapter weights
- Need **multiple checkpoints** without massive storage (125MB total vs 18.8GB)

### ⚠️ Consider Alternatives When:

- Need **higher resolution** (>512×512) - consider SDXL T2I Adapter @ 1024×1024
- Require **extremely precise control** at very low scales (<0.25)
- Working with existing **SDXL pipelines** (use T2I Adapter or ControlNet)

## Limitations

### Current Limitations

- **Resolution**: Trained at 512×512, may not perform optimally at higher resolutions
- **Base model dependency**: Requires FLUX.1-dev base model
- **Grayscale conditioning only**: Trained specifically for brightness/grayscale control
- **Single control type**: Only brightness, not other conditioning types (pose, depth, etc.)

### Recommendations

- For **512×512 generation**, use this Control LoRA
- For **1024×1024+ generation**, consider the SDXL T2I Adapter models
- Experiment with **scales 0.7-1.0** for most use cases
- Use **earlier checkpoints** (625-1250) if you prefer more artistic freedom

## Training Script

```bash
accelerate launch --mixed_precision="bf16" train_control_lora_flux.py \
  --pretrained_model_name_or_path="black-forest-labs/FLUX.1-dev" \
  --dataset_name="<path_to_10k_dataset>" \
  --conditioning_image_column="conditioning_image" \
  --image_column="image" \
  --caption_column="text" \
  --output_dir="./flux-control-lora-brightness-10k" \
  --mixed_precision="bf16" \
  --resolution=512 \
  --learning_rate=1e-4 \
  --proportion_empty_prompts=0.5 \
  --rank=128 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --num_train_epochs=1 \
  --checkpointing_steps=625 \
  --validation_steps=625 \
  --gradient_checkpointing \
  --report_to="wandb"
```

## Available Branches

- `main`: Final model (10,000 samples, recommended)
- `checkpoint-625`: Early checkpoint (2,500 samples)
- `checkpoint-1250`: Mid checkpoint (5,000 samples)
- `checkpoint-1875`: Late checkpoint (7,500 samples)
- `checkpoint-2500`: Final checkpoint (same as main)

## Citation

```bibtex
@misc{flux-control-lora-brightness-10k,
  author = {Oysiyl},
  title = {FLUX.1 Control LoRA - Brightness Control (10k @ 512×512)},
  year = {2025},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k}}
}
```

## Acknowledgments

- Built with [🤗 Diffusers](https://github.com/huggingface/diffusers)
- Base model: [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) by Black Forest Labs
- Dataset: [grayscale_image_aesthetic_3M](https://huggingface.co/datasets/latentcat/grayscale_image_aesthetic_3M) by latentcat
- Training infrastructure: NVIDIA H100 80GB
- LoRA implementation: [PEFT](https://github.com/huggingface/peft) by Hugging Face

## License

Apache 2.0 License. The base FLUX.1-dev model has separate license terms at [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev).