Intrinsic Image Decomposition

Intrinsic image decomposition is the task of separating an image into intrinsic components — decomposing appearance into reflectance (albedo) and shading (illumination), enabling material editing, relighting, and understanding of scene properties independent of lighting conditions.

What Is Intrinsic Image Decomposition?

• Definition: Decompose image into reflectance and shading.
• Input: Single RGB image.
• Output:
• Reflectance (Albedo): Surface color/texture independent of lighting.
• Shading (Illumination): Lighting effects (shadows, highlights).
• Relationship: Image = Reflectance × Shading (in linear space).

Why Intrinsic Decomposition?

• Material Editing: Change surface colors without affecting lighting.
• Relighting: Change lighting while preserving materials.
• Object Recognition: Recognize objects independent of lighting.
• Augmented Reality: Realistic insertion of virtual objects.
• Computational Photography: Advanced photo editing.

Intrinsic Components

Reflectance (Albedo):
- Definition: Intrinsic surface color/texture.
- Properties: Independent of lighting, viewpoint.
- Example: Red ball has red reflectance regardless of lighting.

Shading (Illumination):
- Definition: Lighting effects on surface.
- Components: Direct illumination, shadows, inter-reflections.
- Properties: Depends on lighting, geometry, viewpoint.

Image Formation:

I(x) = R(x) · S(x)

Where:
- I(x): Observed image intensity at pixel x
- R(x): Reflectance (albedo)
- S(x): Shading (illumination)

Intrinsic Decomposition Approaches

Optimization-Based:
- Method: Formulate as energy minimization.
- Energy: Data term + priors (smoothness, sparsity).
- Priors:
- Reflectance is piecewise constant.
- Shading is smooth.
- Reflectance changes at texture edges, shading at geometry edges.
- Examples: Retinex, Intrinsic Images in the Wild.

Learning-Based:
- Method: Neural networks learn decomposition.
- Training: Supervised on synthetic or real data with ground truth.
- Examples: CGIntrinsics, IIW, ShapeNet Intrinsics.
- Benefit: Handle complex real-world images.

Physics-Based:
- Method: Model light transport, inverse rendering.
- Benefit: Physically accurate decomposition.
- Challenge: Requires scene geometry, material properties.

Challenges

Ill-Posed Problem:
- Ambiguity: Infinite (reflectance, shading) pairs can produce same image.
- Example: Dark reflectance + bright shading = bright reflectance + dark shading.
- Solution: Priors, constraints, learning from data.

Texture vs. Shading:
- Problem: Distinguish texture (reflectance) from shading.
- Example: Polka dots (texture) vs. shadows (shading).
- Solution: Multi-scale analysis, learned features.

Complex Lighting:
- Problem: Inter-reflections, subsurface scattering, transparency.
- Challenge: Simple reflectance × shading model insufficient.

Ground Truth:
- Problem: Difficult to obtain ground truth for real images.
- Solution: Synthetic data, multi-illumination capture, crowdsourcing.

Intrinsic Decomposition Methods

Retinex:
- Classic: Separate reflectance and illumination based on gradients.
- Assumption: Reflectance has sharp edges, illumination is smooth.
- Limitation: Oversimplified, doesn't handle complex scenes.

Intrinsic Images in the Wild (IIW):
- Method: Learn from sparse human annotations.
- Annotations: Relative reflectance judgments (same/different material).
- Benefit: Scalable annotation, real-world data.

CGIntrinsics:
- Training: Synthetic data from 3D scenes.
- Network: CNN predicts reflectance and shading.
- Benefit: Large-scale training data.

ShapeNet Intrinsics:
- Training: Rendered 3D objects with known reflectance/shading.
- Benefit: Perfect ground truth for training.

Applications

Material Editing:
- Use: Change surface colors independently of lighting.
- Example: Recolor walls, furniture, clothing.
- Benefit: Realistic edits respecting lighting.

Relighting:
- Use: Change lighting while preserving materials.
- Process: Decompose → modify shading → recompose.
- Example: Change time of day, add/remove lights.

Object Recognition:
- Use: Recognize objects from reflectance (lighting-invariant).
- Benefit: Robust to lighting variations.

Augmented Reality:
- Use: Understand scene lighting for realistic AR.
- Benefit: Virtual objects match real lighting.

Computational Photography:
- Use: Advanced photo editing (selective relighting, material transfer).
- Benefit: Physically plausible edits.

Intrinsic Decomposition Techniques

Multi-Illumination:
- Method: Capture scene under multiple lighting conditions.
- Benefit: Resolve ambiguities, accurate decomposition.
- Challenge: Requires controlled capture.

Multi-View:
- Method: Use multiple viewpoints.
- Benefit: Geometric constraints aid decomposition.

Video:
- Method: Temporal consistency across frames.
- Benefit: More constraints, better decomposition.

Semantic Guidance:
- Method: Use semantic segmentation to guide decomposition.
- Benefit: Material boundaries align with semantic boundaries.

Quality Metrics

MSE (Mean Squared Error):
- Definition: Pixel-wise error in reflectance and shading.
- Limitation: Doesn't account for perceptual quality.

LMSE (Local MSE):
- Definition: MSE after local scaling (handles scale ambiguity).
- Benefit: More robust to global intensity shifts.

DSSIM (Structural Dissimilarity):
- Definition: 1 - SSIM (structural similarity).
- Benefit: Perceptually motivated.

Intrinsic Decomposition Datasets

MIT Intrinsic Images:
- Data: Real objects with ground truth from multi-illumination capture.
- Size: Small but high-quality.

IIW (Intrinsic Images in the Wild):
- Data: Real images with sparse human annotations.
- Size: Large-scale, diverse scenes.

ShapeNet Intrinsics:
- Data: Rendered 3D objects with perfect ground truth.
- Size: Large-scale synthetic data.

MPI Sintel:
- Data: Animated movie frames with ground truth.
- Use: Evaluation on complex scenes.

Future of Intrinsic Decomposition

• Single-Image: Accurate decomposition from single image.
• Real-Time: Fast decomposition for interactive applications.
• Video: Temporally consistent decomposition.
• Semantic: Integrate semantic understanding.
• Physics-Based: Incorporate physical light transport models.
• Generalization: Models that work across diverse scenes.

Intrinsic image decomposition is fundamental to computational photography and computer vision — it enables understanding and manipulating images at the level of materials and lighting, supporting applications from photo editing to augmented reality to object recognition.