Get Started with Lidar Camera Calibrator

The Lidar Camera Calibrator app enables you to interactively perform calibration between a lidar sensor and a camera by estimating a rigid transformation between them.

This topic shows you the Lidar Camera Calibrator app workflow, as well as features you can use to analyze and improve your results. The first and the most important part of the calibration process is to obtain accurate and useful data. For guidelines and tips for capturing data, see Calibration Guidelines.

Prerequisites

Before using the Lidar Camera Calibrator app, you must meet these prerequisites:

• You must have at least four pairs of image and point cloud files. For best calibration results, collect 10–20 data pairs.

  The app supports point cloud data in the PLY and point cloud data (PCD) formats, and images in the PNG, JPG, JPEG, TIF, and TIFF file formats. If your data is stored in a rosbag file, see the Read Lidar and Camera Data from Rosbag File example for information on how to convert it to a supported format.

• The data from the camera and lidar sensors must be time-synchronized.

• Because the app sorts image and point cloud files by filename and pairs them from the sorted lists, to ensure accurate pairings, you must name these files consistently, such as 'img_001' and 'pc_001'.

• When you collect data, the checkerboard must point towards the front axes of the camera (z-axis) and lidar sensor (x-axis). For more information on data collection guidelines, see Calibration Guidelines.

Open Lidar Camera Calibrator App

To open the Lidar Camera Calibrator app, enter this command in the MATLAB^® command prompt.

lidarCameraCalibrator

The app opens to an empty session. Alternatively, you can open the app from the Apps tab, under Image Processing and Computer Vision.

Load Data

The app enables you to load image and point cloud data and specify checkerboard parameters for calibration. You can also load camera intrinsic parameters to perform feature detection.

To load the calibration data into the app, follow these steps:

1. On the app toolstrip, select Import > Import Data, this opens the Import Data dialog box.

2. In the Folder for images field, specify the path to the folder that contains the PNG, JPG, JPEG, TIF, or TIFF image files you want to load. Alternatively, click the folder icon next to the field and navigate to the folder containing the images, then click Select Folder. For example, navigate to this image data, where matlabroot is your MATLAB root folder:

   matlabroot\toolbox\lidar\lidardata\lcc\vlp16\images

3. In the Folder for point clouds field, specify the path to the folder that contains the sequence of PCD or PLY files you want to load. Alternatively, click the folder icon next to the field and navigate to the folder containing the files, then click Select Folder. For example, navigate to this point cloud data, where matlabroot is your MATLAB root folder:

   matlabroot\toolbox\lidar\lidardata\lcc\vlp16\pointCloud

   To view the list of matched pairs, click View Matched Pairs. In the View Matched Pairs dialog box, select or clear the pairs which you want to calibrate. Ensure that the mapping of the data pairs is accurate, and then click Accept.

4. In the Checkerboard Settings section, specify these calibration checkerboard parameters:

   • Units — Select the units of measurement from these options: millimeters, centimeters, meters, or inches.

   • Square Size — Specify the size of each checkerboard square as a positive scalar. For example, specify 81 mm as the square size, in mm, for the data loaded from the vlp16 folder.

   • Padding — Specify the padding values for the checkerboard as a four-element row vector of nonnegative integers. For example, specify [0 0 0 0] mm padding for the data you loaded from vlp16 folder in the previous step. For more information on padding, see Checkerboard Padding.

5. In the Camera Intrinsics section, specify camera intrinsic parameters using one of these options:

   • Load from Workspace — Load camera intrinsic parameters from a valid variable in the MATLAB workspace. The app supports loading camera intrinsic parameters from the cameraIntrinsics and cameraParameters objects.

   • Load from File — Load camera intrinsic parameters from a valid MAT file that stores data using the cameraIntrinsics or cameraParameters object.

   • Compute Intrinsics — Select this option when you want the app to automatically compute camera intrinsic parameters from the input image data.

6. Click OK to import your data.

Tip

To add more images and point clouds to the session at any point in the session, select Import > Add Data to Session.

Detect Features

After you load the image and point cloud data, the app performs an automatic feature detection pass on them. The app detects checkerboard corners from the image data and the checkerboard plane from the point cloud data using the specified checkerboard parameters.

When the app detects checkerboard features in both the image and point cloud, it displays them in the Accepted Data pane, and accepts the image and point cloud data pair for calibration.

When the app does not detect features in the image, the point cloud, or both, it displays them in the Rejected Data pane. You can use the Select Region of Interest or Manually Select Checkerboard Points options to detect checkerboard features in the rejected point cloud data.

Select a data pair from the Accepted Data or Rejected Data pane in the data browser to display it in the visualization pane.

You can use these tools in the Action section of the app toolstrip to work with the accepted and rejected data pairs.

• Move to Accepted Data — Move a data pair from rejected data to accepted data. This movement is possible, only when the data in the Rejected Data browser has valid checkerboard detections for image and point cloud.

• Move to Rejected Data — Move a data pair from accepted data to rejected data.

• Remove — Delete the selected data pair.

For a list of keyboard shortcuts to use in the data browser, see Data Browser.

Change Camera Intrinsic Parameters

To reset camera intrinsic parameters to the default values computed by the app, in the Camera Intrinsics section of the app toolstrip, select Compute Intrinsics. To load different intrinsic parameter values, select Use Fixed Intrinsics, and then Load Intrinsics. In the dialog box that opens, specify the location of your camera intrinsic parameters and load them into the app. You can load intrinsic parameters from a file or from the MATLAB workspace. Then, in the Point Cloud Checkerboard Detection section of the toolstrip, select Detect to re-detect features in the input data using the new intrinsic parameters.

Select Region of Interest

Specifying a region of interest in which to detect features can improve your feature detection results, especially in Rejected Data. To specify a region of interest, first, from the app toolstrip, select Edit ROI. This opens the Edit ROI tab.

In the Edit ROI tab, the app displays point cloud data with an ROI cuboid. Adjust the ROI cuboid to more closely match the region containing the checkerboard. This reduces data rejections and improves performance by containing feature detection to a specific region.

Use these steps to fine-tune the checkerboard detections in your point cloud data using Edit ROI.

1. Select any data pair. You can select a rejected data pair to tune the ROI.

2. To adjust the ROI, in the point cloud pane on the Edit ROI tab, pause on the ROI cuboid. The pointer becomes a hand symbol. Click any side of the ROI and drag to adjust the cuboid size. Ensure the ROI cuboid is at least two meters in each dimension, fully encloses the checkerboard, and contains point cloud data.

   To lock the cuboid dimensions, right click on the cuboid and select Lock Dimensions. You can also undo and redo your modifications to the ROI.

3. Select Snap To ROI to visualize the checkerboard points within the ROI cuboid, and adjust the cuboid size accordingly. To view the whole point cloud, clear Snap To ROI.

   You can also use XY View, YZ View, and XZ View options to visualize top view, front view, and side view of a point cloud scene, respectively.

4. You can update the ROI for each individual frame or for all frames at once.

5. Select Accept to save your changes, or Cancel to discard them.

After updating the ROI for the point cloud frames, on the Calibration tab of the app toolstrip, in the Point Cloud Checkerboard Detection section, select Detect to re-detect features in the input data within the selected ROI.

Tip

Use keyboard shortcuts to perform these tasks more interactively. For Edit ROI keyboard shortcuts, see Edit ROI.

Manually Select Checkerboard Points

If automatic checkerboard detection does not return the desired results, you can improve your results by manually selecting checkerboard points. You can manually select checkerboard points by selecting non-collinear points or a checkerboard region. The app preserves the results of the last selected option.

Select Non-Collinear Points

1. Click Select Points in the Point Cloud Checkerboard Detection section of the app toolstrip. The app opens the Select Points tab, where you can manually select checkerboard points in any point cloud frame.

2. Select any data pair. You can select a rejected data pair and find the checkerboard in the point cloud.

3. Use the zoom and rotate options in the axes toolbar of the point cloud display to locate the checkerboard. For assistance, click Snap To ROI in the Visualize section of the toolstrip to visualize the checkerboard points using the ROI cuboid. You can also use the XY View, YZ View, and XZ View options to visualize the top view, front view, and side view of a point cloud scene, respectively.

4. Click Add Points in the Draw section of the toolstrip, and mark four non-collinear points on the checkerboard plane. The selected point appears in red.

   To clear selected points, click Clear Selection in the Draw section of the toolstrip.

5. After selecting the points, rotate the point cloud to check whether your selection fully encloses the checkerboard and does not contain any background points. You can click Edit Selection in the Edit section of the toolstrip to reposition the polygon vertices. To save your edits, click Save in the Edit section. Otherwise, click Discard to discard your edits.

6. Repeat steps 2–6 for other data pairs.

7. To save the selected points across all data pairs, click Accept in the Close section of the toolstrip. To discard them, click Cancel.

Select Checkerboard Region

1. Click Select Plane in the Point Cloud Checkerboard Detection section of the app toolstrip. The app opens the Select Plane tab, where you can manually select checkerboard points in any point cloud frame.

2. Select any data pair. You can select a rejected data pair and find the checkerboard in the point cloud.

3. Use the zoom and rotate options in the axes toolbar of the point cloud display to locate the checkerboard. You can also use the XY View, YZ View, and XZ View options in the Visualize section of the toolstrip to visualize the top view, front view, and side view of a point cloud scene, respectively.

4. Click Select Checkerboard in the Select section of the toolstrip. Alternatively, you can select Brush Data on the axes toolbar. The pointer changes into a crosshair.

5. Click and drag to create a selection rectangle that highlights the points inside it in red.

6. After selecting the points, rotate the point cloud to check whether any background points have been selected. If your selection contains unwanted points, click Erase in the Select section of the toolstrip to start over.

7. Select Accept to save the selected points, or select Cancel to discard them.

Feature Detection Settings

The app provides these feature detection settings using which you can tune the detection parameters.

• Remove Ground — Remove ground points from the point cloud. The app uses the pcfitplane function to estimate the ground plane. The Remove Ground feature is enabled by default. Select Remove Ground to clear it.

• Cluster Threshold — Clustering threshold for two adjacent points in the point cloud, specified in meters. The clustering process is based on the Euclidean distance between adjacent points. If the distance between two adjacent points is less than the clustering threshold, both points belong to the same cluster. Low-resolution lidar sensors require a higher Cluster Threshold, while high-resolution lidar sensors benefit from a lower Cluster Threshold.

• Dimension Tolerance — Tolerance for uncertainty in the rectangular plane dimensions, specified in the range [0,1]. A higher Dimension Tolerance indicates a more tolerant range for the rectangular plane dimensions.

After updating new detection parameters, select Detect to re-detect features in the input data.

Calibration

When you are satisfied with the detection results, select Calibrate button to calibrate the sensors. If you have an estimated transformation matrix, select Initial Transform to load the transformation matrix from a file or the workspace. The app assumes the rotation angle between the lidar sensor and the camera is in the range [–45 45 ], in degrees, along each axis. For a rotation angle outside this range, use Initial Transform to specify an initial transformation to improve calibration accuracy.

After calibration, the app interface displays the image with the checkerboard points from the point cloud projected onto it. The app uses the projectLidarPointsOnImage function to project the lidar points onto the image. The color information of the images is fused with the point cloud data using the fuseCameraToLidar function.

The app also provides the inaccuracy metrics for the transformation matrix using error plots. The plots specify these errors in each data pair:

• Translation Errors — The difference between the centroid coordinates of the checkerboard planes in the point clouds and those in the corresponding images. The app returns the error values in meters.

• Rotation Errors — The difference between the normal angles defined by the checkerboard planes in the point clouds and those in the corresponding images. The app estimates the plane in the image using the checkerboard corner coordinates. The app returns the error values in degrees.

• Reprojection Error— The difference between the projected (transformed) centroid coordinates of the checkerboard planes from the point clouds and those in the corresponding images. The app returns the error values in pixels.

When you select a data pair in the data browser, the corresponding bars in the error plot are highlighted in dark blue. You can tune the calibration results by removing outliers. Drag the red line on each plot vertically to set error limits. The app selects all the data pairs with an error value greater than error limit as outliers, and highlights the error bars and their corresponding data pairs in the data browser in blue. Right-click any of the selected data pairs on the data browser and select Remove and Recalibrate to delete the outliers and recalibrate the sensors. Deleting outliers can improve calibration accuracy. For a list of keyboard shortcuts to use with the error plots, see Error Plots.

Export Results

You can export the transformation matrix and error metrics, as variables, into the workspace or a MAT-file. You can generate a MATLAB script of the complete app workflow to use in your projects.

Keyboard Shortcuts and Mouse Actions

Note

On Macintosh platforms, use the Command (⌘) key instead of Ctrl.

Use keyboard shortcuts and mouse actions to increase productivity while using the Lidar Camera Calibrator app.

Data Browser

Task	Action
Navigate through data pairs in the Accepted Data or Rejected Data pane	Up or down arrow
Select all data pairs in the data browser	Ctrl+A
Select multiple data pairs above or below the currently selected data pair.	Hold Shift and press the up arrow or down arrow
Select multiple data pairs.	Hold Ctrl and click on data pairs
Delete the selected data pair from the data browser.	PC: Backspace or Delete Mac: delete A dialog box appears to deletion.
Select the data pairN above the currently selected data pair. N is the number of data pairs fully displayed in the data browser at the current time.	PC: Page Up Mac: Hold Fn and press the up arrow
Select the data pairN below the currently selected data pair. N is the number of data pairs fully displayed in the data browser at the current time.	PC: Page Down Mac: Hold Fn and press the down arrow
Select the first data pair in the data browser.	PC: Home Mac: Hold Fn and press the left arrow
Select the last data pair in the data browser.	PC: End Mac: Hold Fn and press the right arrow

Error Plots

Use these shortcuts on the error plots to analyze the data. Operations on any of the three error plots affect the corresponding error bars on all three plots.

Task	Action
Select the error bar left of the currently selected error bar.	Left arrow
Select the error bar right of the currently selected error bar.	Right arrow
Select the error bar right or left of the currently selected error bar, in addition to the currently selected error bar.	Hold Shift and press the left arrow or right arrow
Select multiple error bars.	Hold Ctrl and click error bars
Select all error bars.	Ctrl+A
Delete the selected error bar and corresponding data pair from the data browser. The app then recalibrates the sensors.	PC: Backspace or Delete Mac: delete A dialog box appears to confirm deletion.

Edit ROI

Shortcuts to use on the Edit ROI tab.

Task	Action
Undo ROI size change. Note The app stores only the last three sizes of the ROI, so you cannot undo more than three times in a row.	Ctrl+Z
Redo ROI size change.	Ctrl+Y
Clear ROI selection	Esc

Limitations

The Lidar Camera Calibrator app has these limitations:

• The script generated from Export > Generate MATLAB Script does not include any checkerboard regions manually selected using the Select Checkerboard feature. In the script, the checkerboard region is detected in the specified ROI.

• After manually selecting checkerboard regions using the Select Checkerboard feature, when you return to the Calibration tab, you can see the selected points (highlighted in red) only while viewing the whole point cloud (when Snap To ROI is cleared).