What Is Structure from Motion?

R2026b

Structure from motion (SfM) is the process of estimating the 3-D structure of a scene from a set of 2-D images. SfM is used in many applications, such as 3-D scanning, augmented reality, and visual simultaneous localization and mapping (vSLAM).

Your approach to SfM depends on the number and type of cameras used and whether the images are ordered. If the images are taken with a single calibrated camera, then the 3-D structure and camera motion can only be recovered up to scale. up to scale means that you can rescale the structure and the magnitude of the camera motion and still maintain observations. For example, if you put a camera close to an object, you can see the same image as when you enlarge the object and move the camera far away. If you want to compute the actual scale of the structure and motion in world units, you need additional information, such as:

• The size of an object in the scene

• Information from another sensor, for example, an odometer.

Computer Vision Toolbox™ provides these workflows for structure from motion:

• sfm object — Feature-based reconstruction

• mapAnything object — Deep learning foundation model-based reconstruction

Both produce camera poses and 3-D point clouds. For guidance on choosing between them, see Choose Between SfM and MapAnything.

How sfm Object Implements 3-D Reconstruction

The sfm object implements an incremental SfM pipeline that works with both ordered and unordered image sets. The sfm object requires camera intrinsics, which you can obtain using the Camera Calibrator app or the estimateCameraParameters function. To reconstruct a sparse 3-D point cloud with camera poses, you can use these four object functions in sequence:

1. connectImagePairs — Build an initial view graph by finding visually similar images. This function:

   • Extracts SIFT features from each image.

   • Uses bag-of-words retrieval (DBoW2) to identify candidate image pairs.

   • Matches feature descriptors between candidates using a ratio test.

   • Adds pairs with sufficient matches as edges to the view graph.

   

2. verifyImagePairs — Refine the view graph using geometric constraints. This function:

   • Estimates an essential matrix (five-point algorithm with RANSAC) and a homography in parallel for each connected pair.

   • Selects the model that better explains the correspondences.

   • Removes pairs that fail geometric verification from the view graph.

   

3. triangulateInitialViews — Initialize the 3-D reconstruction by selecting the best image pair from the view graph. This function:

   • Ranks pairs by number of verified matches and spatial distribution.

   • Estimates relative camera pose from the highest-ranked pair.

   • Triangulates initial 3-D points.

   • Evaluates triangulation quality by checking that the median triangulation angle provides sufficient baseline parallax.

4. reconstruct — Incrementally register all remaining views and produce the final reconstruction. For each iteration, this function:

   • Selects the next-best view, which is generally an unprocessed view with the most 2-D-to-3-D correspondences.

   • Solves the Perspective-n-Point (PnP) problem to estimate the camera pose.

   • Triangulates new 3-D points using the estimated pose.

   • Performs local bundle adjustment after each new view.

   • Performs global bundle adjustment after all views are registered to refine all poses and points jointly.

The Structure from Motion from Multiple Views example demonstrates the full multi-view SfM workflow using the sfm object. For parameter tuning guidance, see Best Practices for 3-D Reconstruction Using Structure from Motion.

Build Your Own SfM Pipeline

For more fine-grained control over individual SfM pipeline steps, you can build your own pipeline using the building-block functions that implement each stage independently. Use these functions when the sfm object does not provide sufficient flexibility for your application.

Building an SfM pipeline requires a calibrated camera. Use the Camera Calibrator app or the estimateCameraParameters function to obtain camera intrinsics. With intrinsics in hand, the general SfM workflow is:

1. Find point correspondences between images using feature matching and point tracking.

2. Estimate the relative camera pose from point correspondences.

3. Triangulate 3-D points from point correspondences and camera poses.

4. Refine the reconstruction using bundle adjustment.

If you have two views of a scene, use the two-view reconstruction approach. The Structure from Motion from Two Views example demonstrates a complete two-view SfM workflow, including point tracking and scale recovery. If you have three or more views, use multi-view reconstruction, which extends the two-view approach by computing point tracks across multiple images and iteratively adding views.

Two-View Reconstruction

For the case of two stationary cameras or one moving camera, one view is considered camera 1 and the other camera 2. Camera 1 is assumed to be at the origin with its optical axis along the z-axis.

The first step is to find point correspondences between the two images. For wide baselines where views differ significantly, detect and match features. For narrow baselines with small camera motion between frames, track features across frames. Use these functions to establish correspondences:

• matchFeatures — Match feature descriptors between images (wide baseline).

• vision.PointTracker — Track features across frames (narrow baseline).

Next, estimate the relative pose between the two cameras. The fundamental matrix encodes the epipolar geometry between the two cameras and relates a point in one image to an epipolar line in the other image. From the fundamental matrix, recover the relative rotation and translation of camera 2 in the coordinate system of camera 1. Because the images are taken with a single camera, the location can only be computed up to scale, so the distance between two cameras is set to 1. Use these functions to estimate the relative pose:

• estimateFundamentalMatrix — Estimate the fundamental matrix from point correspondences.

• estrelpose — Recover the relative pose of camera 2 from the fundamental matrix.

Finally, triangulate 3-D points from the matched correspondences and the estimated camera poses. Because the pose is up to scale, the reconstructed structure has the correct shape but not the actual size. Use these functions to triangulate and visualize 3-D points:

• cameraProjection — Compute camera projection matrices from intrinsics and pose.

• triangulate — Triangulate 3-D points from two-view correspondences and camera projection matrices.

• pcshow (Point Cloud Toolbox) and plotCamera — Visualize the reconstructed point cloud and camera poses.

To recover the scale of the reconstruction, you need additional information. One method is to detect an object of a known size in the scene. The Structure from Motion from Two Views example shows how to recover scale by detecting a sphere of a known size in the point cloud.

Multi-View Reconstruction

The approach used for SfM from two views can be extended for multiple views. Multi-view SfM requires point correspondences that span across multiple images, called tracks. Each track corresponds to a single 3-D point observed across several views. A typical approach is to compute the tracks from pairwise point correspondences. Use an imageviewset to manage camera poses and pairwise correspondences, and a worldpointset to store 3-D points and their correspondence to 2-D image points across views.

Once tracks are established, use triangulateMultiview to compute 3-D points from the correspondences spanning multiple views.

Camera pose estimation always contains errors from imprecise point localization, noisy matches, and imprecise calibration. These errors accumulate as the number of views increases, producing drift. One way to reduce the drift is to refine camera poses and 3-D point locations. The nonlinear optimization algorithm bundle adjustment jointly refines camera poses and 3-D point locations to minimize reprojection error. Use these functions for bundle adjustment:

• bundleAdjustment — Jointly refine all poses and 3-D points.

• bundleAdjustmentMotion — Refine poses only (fixed points).

• bundleAdjustmentStructure — Refine 3-D points only (fixed poses).

Another method of reducing drift is by using optimizePoses to perform pose graph optimization over the estimated camera poses when a loop is detected in the trajectory.

Choose Between SfM and MapAnything

The sfm object generally produces more accurate camera poses and 3-D reconstructions on well-textured scenes when it successfully registers all images, but requires careful parameter tuning for each scene type. If the sfm object is unable to register all images through parameter tuning, consider using the mapAnything object. Because the mapAnything object estimates camera poses and scene geometry directly from images using a pretrained feed-forward transformer model, it does not depend on explicit feature matching to register images and handles a large variety of scene types without requiring manual tuning of hyper-parameters. However, using the mapAnything object may result in lower reconstruction accuracy than a successful sfm workflow and also requires GPU acceleration.

Use this table to determine which approach best fits your use case.

Use Case	Recommendation	Rationale
High-accuracy pose estimation on well-textured scenes	sfm	sfm achieves sub-degree rotation and sub-centimeter translation accuracy sfm estimates 10-100x more accurate poses on well-textured scenes
Fast prototyping or no parameter tuning available	mapAnything	mapAnything runs 3-30x faster with predictable runtime on a GPU mapAnything requires only two parameters : BlockSize and OverlapLength sfm has multiple interdependent parameters requiring scene-specific tuning
Camera intrinsics unavailable or uncalibrated images	mapAnything	mapAnything does not require camera intrinsics sfm requires calibrated intrinsics, which is one of the reasons for its higher pose estimation accuracy
CPU-only environment (no GPU available)	sfm	sfm runs on CPU mapAnything requires GPU acceleration
Complete pose recovery for all frames (video stabilization, dense reconstruction)	mapAnything	mapAnything is usually able to register all frames and does not drop views sfm may not register all views on challenging sequences
Low-texture, repetitive, or challenging scenes	mapAnything	mapAnything handles a large variety of scene types without requiring manual tuning of hyper-parameters sfm is brittle on low-texture, forward motion, or repetitive scenes
Object-centric capture (small baselines)	Both viable	Comparable accuracy between both approaches
Large-scale driving or forward motion with sharp turns	Neither ideal Consider Visual SLAM with IMU fusion using monovslam.	sfm registers few frames due to low parallax mapAnything can exhibit cumulative drift on long sequences with sharp turns Visual SLAM with IMU integration helps. For more information, see Performant Monocular Visual-Inertial SLAM.