Reprojected points not matching the checkerboard image

I tried to manually visuallize my multi-cam calibration results and I was stuck at the following issue:

I first visualized the imagePoints extracted from detectCheckerboardPoints onto my image. This step was fine (I could tell that the points were located exactly at the checkerboard intersections).

Then I visualized the reprojected points from the result of estimateMultiCameraParameters. I obtained these points in this path:

multiParams.CameraParameters{:}.ReprojectedPoints(:,:,:)

However, I found the reprojectedPoints were not matching the imagePoints well. I found the reprojected points shared the same origin as the detected points, but had a rotational difference (see image belowm, red dots are imagePoints and blue circles are reprojectdPoints). The reprojection error in this calibration is only 0.4 pixel, with less than 1 pixel for all single images.

Can anyone explain this for me. I appreciate your answer.

[Qu Cao]

Could you share your code and data so that we can repdocue the issue on our end?

[Matt J]

Then I visualized the reprojected points from the result of estimateMultiCameraParameters. I obtained these points in this path: multiParams.CameraParameters{:}.ReprojectedPoints(:,:,:)

That doesn't seem possible. If multiParams is the output you got from estimateMultiCameraParameters, then it would have to be a multiCameraParameters object, which does not have a property called CameraParameters. Also, I can't think of any circumstances where the indenting syntaxmultiParams.CameraParameters{:}.ReprojectedPoints(:,:,:) would have executed without giving you an error message.

[Jiajian]

Hi @Qu Cao, thank you for your help.

I've uploaded one code, one appcode, two matlab data.

MultiCamera_Calibration.m is the end-to-end pipeline code to detect imagepoints, store the intermediate data (source of "Pack.mat"), then use estimateMultiCameraParameters to calcuate the final results (source of "MultiCamParams.mat").

PointsVisualization.mlapp is to visualize the ImagePoints and reprojectedPoints, you should first upload "Pack.mat", then choose the image directory (shared below), then upload "MultiCamParams.mat". With that you can view points on images.

The image directory is here (for privacy concern I only include the first 5x4 views for 4 cameras).

https://drive.google.com/drive/folders/12YhkCYZ7PYnPbe8bHNfGqdbzIj5PrPBW?usp=drive_link

Thank you again for your help. And if you need more information, please contact me through [e-mail]

Best, Jiajian

[Jiajian]

Hi @Matt J, thank you for your comment. The CameraParameters is a private property from multiCameraParameters object. I was able to get it by converting the object to a structure using s = struct(MultiCamParams).

[Jiajian]

@Matt J, also thank you for your second comment. I'm only showing the path of the reprojected points. For sure if I need to visualize, I will need to specify my index, which means I should replace the ":" with my cam numbers or the image index.

Thanks, Jiajian

[Jiajian]

@Qu Cao, there are a few more things I've noticed when using estimateMultiCameraParameters:

1. It does not account for the camera distortion, although I claimed non-zero RadialDistortion in my intrinsics. This is contradictory to the description of its official document: "The function adjusts all image points for distortion according to the lens distortion parameters specified in the intrinsics input."
2. Its CameraPoses outcome is inconsistent with the stereo calibrator app. From one to the other, their rotation matrices are inversed.

Thanks, Jiajian

Answer 1

[Qu Cao]

The problem stems from an incorrect rotationVector assignment in estimateMultiCameraParameters. To fix the issue, please change Line 111 in estimateMultiCameraParameters

from

rotationVectors(i, :) = rotmat2vec3d(patternExtrinsics(viewIdx, camIdx).R');

to

rotationVectors(i, :) = rotmat2vec3d(patternExtrinsics(viewIdx, camIdx).R);

Note that the transpose should be removed.