Scene Flow Estimation

Scene flow estimation is the task of predicting 3D motion vectors for scene points over time, extending optical flow from image-plane displacement to physical-space dynamics - it combines geometry and motion to model real-world movement in x, y, and z dimensions.

What Is Scene Flow?

• Definition: Dense 3D motion field estimated from stereo, RGB-D, or multi-view temporal input.
• Difference from Optical Flow: Optical flow gives 2D image displacement only.
• Required Signals: Depth and camera geometry are needed to recover true 3D movement.
• Output Usage: Dynamic scene understanding for robotics and autonomous driving.

Why Scene Flow Matters

• Physical Motion Awareness: Captures forward and backward depth movement, not just lateral pixel shift.
• Planning Support: Better inputs for collision prediction and trajectory planning.
• 3D Tracking: Improves object motion estimation in world coordinates.
• Sensor Fusion Value: Bridges camera and depth modalities in one representation.
• High-Stakes Utility: Critical for safety-sensitive perception stacks.

Scene Flow Pipeline

Geometry Estimation:
- Recover depth or disparity from stereo or depth sensor.
- Convert pixels to 3D point representations.

Temporal Correspondence:
- Match points across time with learned or geometric correspondence methods.
- Estimate 3D displacement vectors per point.

Consistency Regularization:
- Enforce geometric and temporal consistency constraints.
- Reduce noise and occlusion-induced errors.

How It Works

Step 1:
- Compute frame-wise geometry and extract point or voxel features.

Step 2:
- Predict point correspondences and 3D displacement, then refine with consistency losses.

Scene flow estimation is the 3D motion representation that turns image dynamics into physically meaningful movement understanding - it is essential when systems must reason in real-world coordinates, not only image space.