README.md

---
license: other
base_model: black-forest-labs/FLUX.1-dev
tags: [flux, lora, style, character-design, concept-art, comfyui]
---

# art-lora-flux1-dev

Style LoRA for **FLUX.1-dev**. Trained on 19 curated painterly character-design references.

**Trigger:** `sk3tchref`

Hand-painted character concept art — broad flat brush shapes, hard-edged colour blocking,
minimal blending, muted olive / charcoal / cream / tan palette, flat off-white background
with a thin shadow smear, stylized elongated proportions.

![samples](samples/_grids/grid2000.jpg)

---

## Repo contents

| Path | What it is |
|---|---|
| `sk3tchref.safetensors` | **Final weights (step 2000) — use this one** |
| `sk3tchref-step000005/1000/1500/2000.safetensors` | Intermediate checkpoints |
| `workflow.json` | ComfyUI workflow — drag onto the canvas |
| `workflow_api.json` | Same graph in API format, for `POST /prompt` |
| `dataset/` | All 19 training images **and** their caption files |
| `samples/` | Outputs per checkpoint + comparison grids |

Style LoRAs overfit, so the best checkpoint is not always the last. All of them are here
so you can compare — see `samples/_grids/`.

---

## 1. Download the models

Everything below goes into your ComfyUI `models/` tree. Total ~34 GB.

```bash
cd /workspace/ComfyUI      # adjust to your ComfyUI root
mkdir -p models/diffusion_models models/text_encoders models/vae models/loras
```

### Transformer — FLUX.1-dev, 23.8 GB

The official repo **`black-forest-labs/FLUX.1-dev` is gated**: you must accept the licence
on the model page and use a read token. Same for `FLUX.1-schnell`.

```bash
# Option A — official, gated. Accept the licence first at
#   https://huggingface.co/black-forest-labs/FLUX.1-dev
# then create a READ token at https://huggingface.co/settings/tokens
export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx

wget --header="Authorization: Bearer $HF_TOKEN" \
  -O models/diffusion_models/flux1-dev.safetensors \
  "https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors"
```

```bash
# Option B — Comfy-Org mirror. Identical weights, no token needed at time of writing.
wget -O models/diffusion_models/flux1-dev.safetensors \
  "https://huggingface.co/Comfy-Org/flux1-dev/resolve/main/flux1-dev.safetensors"
```

Option B is what this LoRA was trained against. Mirrors can be gated or removed without
notice — if Option B starts returning `401`, fall back to Option A.

### Text encoders — 10 GB

```bash
wget -O models/text_encoders/t5xxl_fp16.safetensors \
  "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp16.safetensors"

wget -O models/text_encoders/clip_l.safetensors \
  "https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors"
```

### VAE — 335 MB

```bash
# ungated mirror (black-forest-labs/FLUX.1-schnell is gated and returns 401)
wget -O models/vae/ae.safetensors \
  "https://huggingface.co/ffxvs/vae-flux/resolve/main/ae.safetensors"
```

### This LoRA

```bash
wget -O models/loras/sk3tchref.safetensors \
  "https://huggingface.co/kirusanth08/art-lora-flux1-dev/resolve/main/sk3tchref.safetensors"

# intermediate checkpoints, if you want to compare
for S in 00000500 00001000 00001500 00002000; do
  wget -O "models/loras/sk3tchref-step$S.safetensors" \
    "https://huggingface.co/kirusanth08/art-lora-flux1-dev/resolve/main/sk3tchref-step$S.safetensors"
done
```

If the repo is private, add `--header="Authorization: Bearer $HF_TOKEN"` to those too.

### Verify

A truncated download does not error — it fails later at load time with
`'NoneType' object has no attribute 'Params'`. Check the sizes:

```bash
ls -l models/diffusion_models/flux1-dev.safetensors   # 23802932552
ls -l models/text_encoders/t5xxl_fp16.safetensors     #  9787841024
ls -l models/text_encoders/clip_l.safetensors         #   246144152
ls -l models/vae/ae.safetensors                       #   335304388
```

---

## 2. Run it

```bash
wget -O workflow.json \
  "https://huggingface.co/kirusanth08/art-lora-flux1-dev/resolve/main/workflow.json"
```

Drag `workflow.json` onto the ComfyUI canvas. It wires up:

```
UNETLoader (flux1-dev, fp8_e4m3fn)
  └─ LoraLoaderModelOnly (sk3tchref.safetensors, strength 1.0)
       └─ KSampler ── VAEDecode ── SaveImage
DualCLIPLoader (clip_l + t5xxl_fp16, type "flux")
  └─ CLIPTextEncode ─┬─ FluxGuidance 3.0 ──→ positive
                     └─ ConditioningZeroOut ─→ negative
```

`workflow_api.json` is the same graph for `POST /prompt`.

### Settings that matter

**`cfg` must stay at `1.0`.** Flux dev is CFG-distilled — raising it destroys the image,
and negative prompts do nothing, which is why the negative branch is just a zeroed-out
copy of the positive. `FluxGuidance` (~3.0) is the real prompt-adherence knob.

Sampler `euler`, scheduler `beta`, 28 steps, 832×1216.

### Prompt template

```
sk3tchref, a full body character illustration of <SUBJECT>, hand-painted character
concept art, broad flat brush shapes, hard-edged colour blocking, muted desaturated
palette, flat off-white background, thin grey shadow smear under the feet
```

The trigger must come first.

### Strength guide

| Strength | Result |
|---|---|
| 0.6 | Too weak — the base model's smooth prior shows through |
| 0.8 | Works |
| **1.0** | **Recommended — strongest read, no subject bleed** |
| 1.1+ | Training subjects and poses start reappearing |

See `samples/final/sweep.jpg`.

---

## 3. How it was trained

kohya `sd-scripts` (`sd3` branch), `flux_train_network.py`, on an RTX 5090 32 GB.
~1.75 s/it, ~1 hour for 2000 steps.

```bash
accelerate launch --num_processes 1 --mixed_precision bf16 \
  --num_cpu_threads_per_process 8 flux_train_network.py \
  --pretrained_model_name_or_path $M/diffusion_models/flux1-dev.safetensors \
  --clip_l $M/text_encoders/clip_l.safetensors \
  --t5xxl $M/text_encoders/t5xxl_fp16.safetensors \
  --ae $M/vae/ae.safetensors \
  --dataset_config dataset.toml \
  --output_dir ./out --output_name sk3tchref \
  --save_model_as safetensors --save_precision bf16 \
  --network_module networks.lora_flux --network_dim 24 --network_alpha 24 \
  --network_train_unet_only \
  --optimizer_type adamw8bit --learning_rate 1e-4 \
  --lr_scheduler constant_with_warmup --lr_warmup_steps 40 \
  --max_train_steps 2000 --save_every_n_steps 500 \
  --gradient_checkpointing --mixed_precision bf16 --fp8_base --sdpa --highvram \
  --cache_latents_to_disk --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk \
  --max_data_loader_n_workers 2 --persistent_data_loader_workers --seed 42 \
  --timestep_sampling shift --discrete_flow_shift 3.1582 \
  --model_prediction_type raw --guidance_scale 1.0 --loss_type l2
```

Dataset config: aspect-ratio bucketing at 1024 base, `min_bucket_reso 512`,
`max_bucket_reso 1536`, `keep_tokens = 1`, `shuffle_caption = false`.

---

## 4. The captioning method

This is the part that decides whether a style LoRA works.

Captions describe **content only** — who the character is, what they wear, what they do:

```
sk3tchref, two views, front and back, of a heavy-set older man with a grey beard in an
olive green jacket, cream shirt and dark green trousers, holding a bottle
```

Note what is *absent*: no "painterly", no "flat colour blocking", no "muted palette", no
"white background". Everything constant across the set and missing from the captions
collapses onto the single token they all share — `sk3tchref`.

Write style words into the captions and the style scatters across those common words
instead, producing a LoRA that is weak, hard to trigger, and unresponsive to strength.
`keep_tokens = 1` pins the trigger at the front so shuffling never displaces it.

Every caption in `dataset/` was written by looking at the image.

---

## Why a LoRA and not `--sref`

Reference-image conditioning was tested first and does not work for style transfer:

| Approach | Result |
|---|---|
| Redux / IPAdapter | Encodes the reference into **content** tokens. Below ~0.2 strength nothing transfers; above it the reference's *subject* appears — its hat, its props, its character. No setting yields brushwork without subject bleed. |
| Flux.2 Klein / Kontext edit-refs | Trained to **preserve** reference content; "style only" fights the objective. |
| Prompt engineering alone | Yields Flux's own house style. Sweeping guidance moves composition, not paint. |

Conditioning transfers content. Prompting transfers the base model's prior. Only weight
updates transfer style.