Controlnet

ControlNet and Image Conditioning

What is ControlNet?
ControlNet adds spatial conditioning to diffusion models, allowing precise control over generated images using edge maps, poses, depth maps, and more.

Control Types
| Control | Input | Use Case |
|---------|-------|----------|
| Canny Edge | Edge detection | Preserve structure |
| Pose | OpenPose skeleton | Character poses |
| Depth | Depth map | 3D-aware generation |
| Segmentation | Semantic masks | Layout control |
| Normal Map | Surface normals | Lighting/texture |
| Scribble | Hand-drawn lines | Sketch to image |
| LineArt | Line drawings | Illustration style |

Basic Usage

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import cv2
import numpy as np

# Load ControlNet
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet
)

# Prepare control image
image = cv2.imread("input.jpg")
edges = cv2.Canny(image, 100, 200)

# Generate
result = pipe(
    prompt="a detailed architectural rendering",
    image=edges,
    num_inference_steps=30
).images[0]

Multi-ControlNet
Combine multiple controls:

controlnets = [
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny"),
    ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth")
]

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    model_id,
    controlnet=controlnets
)

result = pipe(
    prompt="...",
    image=[edge_image, depth_image],
    controlnet_conditioning_scale=[1.0, 0.8]
)

IP-Adapter
Control generation with reference images:

# Use reference image to guide style/content
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models")
result = pipe(
    prompt="a dog in the park",
    ip_adapter_image=reference_image  # Style reference
).images[0]

Use Cases
| Use Case | Controls |
|----------|----------|
| Architecture | Canny + Depth |
| Character design | Pose + Reference |
| Product visualization | Depth + Segmentation |
| Before/after edits | Canny (preserve structure) |

Best Practices
- Match control strength to desired fidelity
- Preprocess control images consistently
- Combine controls for more precise output
- Use lower strength for creative freedom