Labeling Lidar Point Clouds for Object Detection
Use the Lidar Labeler app in Lidar Toolbox™ to label lidar point clouds for object detection—an essential step in a deep learning workflow for autonomous driving and robotics applications. The app enables manual labeling of objects in the point cloud with oriented bounding boxes and labeling automation using inbuilt and custom automation algorithms.
Published: 7 Nov 2021
Labeling point clouds is one of the most tedious tasks in deep learning workflows for LiDAR. So MATLAB is providing a LiDAR Labeler app in LiDAR toolbox which alleviates some of the most common pain points of point cloud labeling. In this video, I'll show you how to label your point cloud data using LiDAR Labeler app for applications like autonomous driving, robotics, and augmented reality.
We'll first see how to ground truth label point cloud for object detection. And then, we'll go over how to automate the labeling process. Let's start by opening the app from App menu. We can find the LiDAR Labeler app under Image Processing and Computer Vision section. Alternatively, we can open the app from command line by typing LiDAR Labeler. We'll first load the point cloud into the app by clicking Add Point Clouds. You can also import labels from files, from Workspace, or you can open a session you have already saved before.
We can load raw point clouds in different file formats, like PCD, VLY, PCAP, LAS, and LAZ. Here, we are loading sequential point clouds off a highway driving scenario in PCD format. Once the point cloud sequence is loaded, as a best practice, we can play the entire sequence to get an overview about the scenario and objects in the point cloud sequence. There are some other features that will help you to get a better understanding about the point cloud before labeling.
For example, you can hide the ground points using Range-based Floodfill or Fit Ground Plane methods. You can also use the Snap to Cluster feature to apply different types of clustering on the point cloud data. Let's run the sequence again now. Now that we know the objects we need to label in the sequence, we can define them by clicking Label from the ROI Label tab in the left pane. Let's give the label name as Car and treat cuboid over the cars in the scene.
We can select this Shrink to Fit option so that the cuboids fit properly over the object clusters. Now, let's label the car in the sequence by clicking on the car cluster in the point cloud. We can use Projected View option to see different views of the selected objects and then manually fit bounding box over the object. The ROI feature in the app enables you to define the ROI and label objects in the ROI. You can change the size of points in the point cloud to visualize the clusters.
Another useful feature in the app is Camera View option. You can save and reuse custom camera views of the point cloud data. Now that we have seen how to label objects in the point cloud, let's talk a little bit about automating the labeling process. For that, we'll go to the Label tab and select the Automation Algorithm. In this case, let's select the Point Cloud Temporal Interpolator. This algorithm estimate cuboids in the intermediate frames using interpolation between the cuboid arrows in the key frames.
We'll then click the Automate button and manually label some key frames. Here, we have selected five random instances as key frames. After this, we'll select the Automate tab and click Run. Now you can see that all the intermediate frames are labeled automatically. You can go through the label frames and can manually edit any of the frames as per needed.
Finally, you can accept the automation and export the ground truth label point cloud to MATLAB workspace or to a file. Alternatively, if you have a custom automation algorithm, you can use them by selecting Add Algorithm option. We can then import the algorithm.
We are following this workflow example to demonstrate how to use a pre-trained object detection network to automate your LiDAR labeling process. Here, we are using a pre-trained point pillar object detection network that can fit oriented bounding boxes around vehicles in the point cloud data. Once you have imported the automation algorithm, you can find it under Select Algorithm panel. Now, click the Automate button, followed by the Run button.
You can now see the bounding boxes around the detected objects. If the detection seems fine, we can accept the automation and export the ground truth label point cloud to MATLAB workspace or to a file as we've done before. Here, I am exporting the labels to MATLAB workspace. A quick look at the data source will show us the label properties, like source name, signal name, signal types, et cetera. We can further use the label data set to train an object detection network on your LiDAR point cloud data.
Follow the links below to learn more. If you have any questions or comments, please let us know.