Is It Possible to Get Depth from a Single Image? How Apple's Depth Pro Estimates Depth

Special thanks to those who inspired and supported this post:
Murat Gülhan, Muharrem Aytekin, and Hakan Yıldırım

Introduction: The Challenge of Depth from a Single Image

Depth estimation from a single image is one of the most fascinating challenges in computer vision. Unlike humans who can perceive depth through various visual cues, computers struggle to understand the three-dimensional world from a flat, two-dimensional image.

3D World Coordinates vs. 2D Image Coordinates

In computer vision, we often represent points in the 3D world and their projections onto a 2D image using matrices:

3D World Point (Homogeneous Coordinates)

A point in 3D space is represented as a column vector:

Reference: Tola, Engin. (2005). Multi-view 3D Reconstruction of a Scene Containing Independently Moving Objects.

Where:

• X, Y, Z: 3D world coordinates
• 1: Homogeneous coordinate (for matrix operations)

In homogeneous coordinates, this is represented as:

2D Image Point (Homogeneous Coordinates)

A point in the 2D image is represented as a column vector:

Where:

• u, v: 2D image coordinates (pixels)
• 1: Homogeneous coordinate

Projection Matrix (P)

A camera projects a 3D point onto a 2D image plane. This is represented with a projection matrix:

Where:

• P: 3×4 projection matrix that combines intrinsic and extrinsic parameters
• f_x, f_y: Focal lengths in pixels
• c_x, c_y: Principal point coordinates
• Last column: All zeros for perspective projection (no translation in image plane)

The projection matrix P = K × [R | t] combines both the camera's intrinsic parameters (K) and extrinsic parameters (rotation and translation) into a single 3×4 matrix that directly maps 3D world points to 2D image coordinates.

---

Why Single-Image Depth is Challenging

Estimating depth from a single image is fundamentally challenging because, unlike stereo vision (which uses two images from slightly different viewpoints), a single image does not provide direct geometric cues about how far objects are from the camera. The 3D world is projected onto a 2D plane, causing loss of depth information.

u = f_x × (X_c / Z_c) + c_x,    v = f_y × (Y_c / Z_c) + c_y

Here, (X_c, Y_c, Z_c) are the 3D coordinates of a point in the camera coordinate system, (u, v) are the corresponding pixel coordinates in the image, f_x and f_y are the focal lengths in pixels, and (c_x, c_y) are the principal point offsets.

Why can't we get depth from a single image?

From a single image, we only observe (u, v), but both X_c and Z_c (or Y_c and Z_c) are unknown. This means that there are infinitely many 3D points that could correspond to the same pixel; therefore, the depth Z_c cannot be determined directly.

Stereo Cameras for More Accurate Depth

While single-image depth estimation is impressive, using two cameras—known as a stereo camera setup—can provide much more accurate depth information. Stereo vision mimics human binocular vision: two cameras are placed a fixed distance apart (the "baseline") and capture images of the same scene from slightly different viewpoints.

How can we estimate depth with two images (stereo)?

With two images taken from slightly different viewpoints (a stereo pair), we can find the same point in both images and measure its horizontal displacement, called disparity (d). For rectified stereo images, the depth Z_c can be computed as:

---

Stereo Geometry: Depth Calculation

Consider two cameras: left (L) and right (R), separated by a distance called the baseline (B).

• The left camera center: O_L
• The right camera center: O_R, shifted by B along the x-axis.

For a 3D point:

• In the left camera: X_L = X,    Z_c = Z
• In the right camera: X_R = X - B,    Z_c = Z

Image projections:

• Left image: u_L = f × (X / Z)
• Right image: u_R = f × ((X - B) / Z)

Disparity (d): The horizontal difference between the same point in both images:

d = u_L - u_R = (f × B) / Z

Solving for depth (Z):

Z = (f × B) / d

This formula summarizes how stereo cameras compute real depth.

---

Z_c = (f × B) / d

where:

• f is the focal length (in pixels),
• B is the baseline (distance between the two camera centers),
• d is the disparity (difference in u coordinates between the left and right images).

By matching corresponding points and measuring disparity, we can triangulate the true depth of each point in the scene.

How Stereo Depth Works:

• By comparing the positions of the same feature in both images (called "disparity"), we can triangulate the actual distance to that feature.
• The greater the disparity (difference in position), the closer the object is to the cameras.
• This geometric approach provides direct, metric depth measurements, unlike single-image methods that rely on learned cues and priors.

---

Alternative: Depth from Multiple Frames (Structure from Motion)

While stereo cameras use two lenses to capture depth, you can also estimate depth using a single moving camera by capturing multiple frames from different viewpoints. This technique is known as Structure from Motion (SfM) or Multi-View Geometry.

• Key idea: If you move a camera (for example, by walking with your phone or flying a drone), each frame provides a slightly different view of the scene. By tracking how features move between frames, you can triangulate their 3D positions—just like with two cameras, but using time-separated images instead of two simultaneous ones.
• Depth from motion: The more the camera moves (with known motion), the more accurate the depth estimation can be, as long as there is enough parallax (change in viewpoint).

How it works:

• Detect and track feature points across multiple frames.
• Estimate the camera's motion between frames (using visual odometry or IMU data).
• Triangulate the 3D position of each feature using the known camera poses and the observed 2D positions.

Applications:

• Used in 3D Mapping, AR/VR, and robotics.

---

Depth Estimation with Drones

Drones are a practical example of using multiple frames for depth estimation:

• As a drone flies, its onboard camera captures a sequence of images from different positions.
• By analyzing how features move across these frames, the drone can build a 3D map of its environment.
• This is crucial for tasks like obstacle avoidance, autonomous navigation, and mapping.

Advantages:

• No need for two cameras—just a single camera and motion.
• Works in environments where stereo setups are impractical.

---

Apple's Depth Pro Model: Zero-Shot Metric Monocular Depth Estimation

Apple's Depth Pro is a new foundation model for zero-shot metric monocular depth estimation. In other words, given a single RGB image, Depth Pro produces a high-resolution, per-pixel depth map in real-world units (meters)—without needing any camera metadata.

Key highlights:

• No camera metadata required: Depth Pro can infer the camera's field-of-view (focal length) directly from the image, so it works even when EXIF data is missing.
• High speed and resolution: On modern GPUs, it processes a 1536×1536 image (~2.3 MP) in about 0.3 seconds, producing sharp, high-frequency detail (e.g., fine hair and object edges).
• Metric depth (with caveats): Produces depth maps in real-world units (meters), but the absolute accuracy can vary depending on scene type, camera properties, and image content (see discussion below).

Core Innovations

• Multi-scale Vision Transformer (ViT) Architecture:
  Depth Pro uses an efficient, patch-based ViT backbone that fuses information at multiple scales for dense, high-resolution prediction.

• Focal-Length Estimation Head:
  A dedicated network head predicts the camera's focal length (field-of-view) from the image itself, enabling metric depth estimation without any external camera parameters.

Training Objectives

Depth Pro predicts canonical inverse depth (1/distance) which emphasizes nearby objects. The main loss function (LMAE) measures prediction accuracy while ignoring the worst 20% of pixels in real-world datasets.

Additional loss functions for sharp boundaries:

• LMAGE: Measures edge accuracy
• LMALE: Measures smoothness
• LMSGE: Measures edge precision

This approach produces depth maps with sharp details and smooth transitions.

Training Curriculum

Depth Pro uses a two-stage approach:

Stage 1: Train on mixed datasets (real + synthetic) using LMAE loss for robustness Stage 2: Train only on synthetic data using multiple loss functions (LMAE, LMAGE, LMALE, LMSGE) for sharp boundaries

Evaluation Metrics for Sharp Boundaries

Uses existing annotations (segmentation, saliency, matting) to measure boundary sharpness:

• Compares neighboring pixels for depth differences (5-25% thresholds)
• Calculates precision/recall vs. ground truth
• No manual annotation needed and scale-invariant

This ensures sharp, accurate boundaries for real-world applications.

Focal Length Estimation

How does Depth Pro handle missing camera information?

Since many images lack accurate EXIF metadata (camera information), Depth Pro includes a focal length estimation head that predicts the camera's field of view directly from the image content.

How It Works

• A small convolutional network analyzes the image to predict the horizontal angular field-of-view
• Uses frozen features from the depth estimation network plus task-specific features from a separate ViT encoder
• Trained separately from the main depth network to avoid balancing multiple objectives