Use an LM iterative method to iteratively optimize the transformation matrix between two video frames in video image stitching.

When stitching two video images (frames), after obtaining the transformation matrix by stitching two images from the previous frame, how can you use the LM algorithm to optimize the transformation matrix based on the feature points of the two images from the subsequent frame, so that the two images of the subsequent frame are also stitched together? Please provide a code reference, assuming that the transformation matrix and feature points from the previous frame are known.Thank you very much.

Answer 1

[Simar]

Hello 兴炎 孙,

As per my understanding to improve video image stitching by combining frames and fine-tuning the transformation matrix, the Levenberg-Marquardt (LM) algorithm in MATLAB is a recommended method. This approach requires the Computer Vision Toolbox for its key capabilities in feature detection, extraction, and matching. Steps to Optimize Transformation Matrix are as follows:

Step 1: Feature Matching (Given as mentioned)

First, identify matched feature points (matchedPoints1 and matchedPoints2) in two consecutive frames along with an initial transformation matrix (initialTransform). Utilize functions such as “detectSURFFeatures”, “extractFeatures”, and “matchFeatures” for this purpose.

Step 2: Objective Function

Create an objective function to measure the difference between the actual locations of matched points in the second frame and their estimated locations. This estimation comes from applying the transformation matrix to the first frame’s matched points.

function residuals = objectiveFunction(x, matchedPoints1, matchedPoints2)
T = reshape(x, 3, 3);
pointsTransformed = transformPointsForward(projective2d(T), matchedPoints1);
residuals = pointsTransformed - matchedPoints2;
residuals = residuals(:);
end

Step 3: Optimization with Levenberg-Marquardt

Use MATLAB “lsqnonlin” for optimization with the objective function, initial guess from transformation matrix, and specified options.

initialGuess = initialTransform(:);
options = optimoptions('lsqnonlin', 'Algorithm', 'levenberg-marquardt', 'Display', 'iter');
[optimizedParams,~,residual,exitflag,output] = lsqnonlin(@(x) objectiveFunction(x, matchedPoints1.Location, matchedPoints2.Location), initialGuess, [], [], options);
optimizedTransform = reshape(optimizedParams, 3, 3);

Important Considerations:

• Ensure initial transformation matrix is correctly formatted.
• Quality of the initial guess impacts optimization results.
• This example focuses on projective transformations; adjust accordingly for other types.

Please refer to the following documentation links-

• Computer Vision Toolbox- https://www.mathworks.com/help/vision/index.html?searchHighlight=COMPUTER%20VISION%20TOOLBOX&s_tid=srchtitle_support_results_1_COMPUTER%20VISION%20TOOLBOX
• reshape- https://www.mathworks.com/help/matlab/ref/reshape.html?searchHighlight=reshape&s_tid=srchtitle_support_results_1_reshape
• Isqnonlin- https://www.mathworks.com/help/optim/ug/lsqnonlin.html
• optimoptions- https://in.mathworks.com/help/optim/ug/optim.problemdef.optimizationproblem.optimoptions.html?searchHighlight=optimoptions&s_tid=srchtitle_support_results_1_optimoptions
• detectSURFFeatures- https://www.mathworks.com/help/vision/ref/detectsurffeatures.html?searchHighlight=detectsurfeatures&s_tid=srchtitle_support_results_1_detectsurfeatures
• extractFeatures- https://www.mathworks.com/help/vision/ref/extractfeatures.html?searchHighlight=extractfeatures&s_tid=srchtitle_support_results_1_extractfeatures
• matchFeatures- https://www.mathworks.com/help/vision/ref/matchfeatures.html?searchHighlight=matchfeatures&s_tid=srchtitle_support_results_1_matchfeatures

Best wishes,

Simar