Deep Learning for Lidar
Learn what deep learning for lidar is and how to apply it for object detection and semantic segmentation using MATLAB. We start by explaining the basics and different networks available. Then, we demonstrate how to apply deep learning on lidar data using the PointPillars network for an object detection workflow.
Published: 5 Jul 2024
Hello, everyone. In this video, we are going to explore deep learning for Lidar. We look at what is deep learning in context of Lidar, what are the different deep learning models suitable for Lidar data, and finally, how to apply these models for tasks such as classification, detection, and segmentation in MATLAB. Let's get started.
Deep learning is a subset of machine learning that uses layered neural networks to analyze and learn from large amounts of data. Deep learning can be used on Lidar data to perceive the environment in autonomous driving, aerial, and robotics applications by identifying and classifying objects in the scene.
For automated driving, we can use SalsaNext, RandLA-Net, and SqueezeSegV2 for semantic segmentation. Each of these networks has its own benefits that makes it best fit for a specific kind of data. For instance, SalsaNext provides better accuracy in automated driving applications than RandLA-Net. But RandLA-Net can be applied both on organized and unorganized point clouds, while the former can be applied only on organized point clouds.
For object detection in automated driving applications, we can use PointPillar network or Complex YOLO v4. For lane detection, we can use K-Lane network. And when we have aerial Lidar data for applications like surveillance, agriculture, and urban planning, we can use RandLA-Net and PointNet++ networks for semantic segmentation. We can also integrate an external deep learning models from TensorFlow, ONNX, or PyTorch into a MATLAB workflow. Please refer to this documentation to learn more about this.
Now that we have seen these different networks, let's see how to implement them for an application in MATLAB. For this, we can use an example of object detection for automated driving applications using PointPillar network. PointPillar networks addresses some of the common challenges in training robust detectors, like sparsity of data per object, object occlusions, and sensor noise by an encoder that can learn representation of point clouds organized in vertical columns called pillars.
We'll start with loading the data, followed by preprocessing the loaded data set to split them into training and testing data set. We can also add an optional step of data augmentation to help the model improve accuracy of its prediction. Once the data is ready, we can create a PointPillar object detector network using pointPillarsObjectDetector function. This function requires you to specify inputs that parameterize the PointPillars networks, like class names, anchor boxes, point cloud range, and Voxel size.
After that, we can take two roads. We can use a pretrained network from MATLAB to detect cars and trucks in your point cloud. Alternatively, you can take the route of transfer learning. Transfer learning is a common deep learning technique in which you take a pretrained network as a starting point to train a network for a new task. To perform transfer learning with a pretrained point pillar object detector network, specify the classes and their corresponding anchor boxes. Then train the network on a new data set.
We can train the network on a labeled data by specifying the network training parameters using trainingOptions function. And then use this function to train point pillars object detector. But if we don't have a label point cloud data set, we can use the Lidar Labeler app to label point cloud data for object detection. This app also allows us to label point cloud data for applications like semantic segmentation and late detection of ground and aerial Lidar data.
We can export these labels into MATLAB workspace and use them for training as shown before. Please refer to this video series to learn more about Lidar Labeler app. Now that we have trained the network, we can finally use this trained network to detect cars and trucks from the point cloud data using detect function. Here is how our data looks like.
Please note that we can use the same workflow to implement any other deep learning models in MATLAB. So in summary, we have seen how to use deep learning models available in MATLAB to detect segment and classify point cloud data. We also saw that we can use Lidar Labeler app to label point cloud data that can be then used to train our networks. To learn more about this, please refer to our documentation. Thank you.