Aerial Lidar Processing in MATLAB
Overview
Introduction of low cost lidar sensors has increased adoption of lidar workflows in various aerial applications such as mapping, surveying, inspection and monitoring. High accuracy and high density of the lidar data renders it useful in space management, security, and defense applications.
Highlights
In this webinar, you will learn how to develop complex lidar processing algorithms. We will walk through a workflow example and address common challenges in the process, such as
- Reading and processing large lidar point clouds
- Distortion and tracking errors due to motion
- Labeling huge datasets for AI workflows
About the Presenter
Minhaj Palakkaparambil Mohammed is a product manager at MathWorks, with a focus on autonomous systems and lidar point cloud processing. Prior to joining MathWorks he worked as a lead engineer for developing autonomous systems. He holds a masters degree from NIT Suratkal in India.
Rishu Gupta is a senior application engineer at MathWorks India. He primarily focuses on automated driving, aerial, and deep learning applications. Rishu has over nine years of experience working on applications related to visual contents. He previously worked as a scientist at LG Soft India in the Research and Development unit. He has published and reviewed papers in multiple peer-reviewed conferences and journals. Rishu holds a bachelor’s degree in electronics and communication engineering from BIET Jhansi, a master’s in visual contents from Dongseo University, South Korea, working on the application of computer vision, and a Ph.D. in electrical engineering from University Technology Petronas, Malaysia with focus on biomedical image processing for ultrasound images.
Recorded: 16 Mar 2022
Hello, everyone. Welcome to today's webinar on aerial lidar processing. My name is Minhaj Falake. I'm a product manager at MathWorks responsible for Lidar Toolbox. I've been at MathWorks for the past three years supporting 3D perception for a variety of application areas like robotics, autonomous systems, aerospace, and defense.
Prior to that, I worked in a startup as a lead engineer developing perception systems for autonomous driving applications. With me today, we have Dr. Rishu Gupta, a Senior Application Engineer from MathWorks India. Rishu, can you please introduce yourself before going to the session?
Thank you, Minhaj. Hello, everyone. My name is Rishu Gupta, and I work as a senior application engineer at MathWorks. I focus on application areas related to autonomous system development and perception algorithms. I have been associated with MathWorks for almost five years now.
Thanks, Rishu. So without further ado, let's get started. We are going to discuss three things in our agenda today. First one is an insight into why and where lidar sensors are used. Secondly, we'll discuss a few common challenges our users face while processing aerial lidar data. Finally, we'll demonstrate how to segment aerial lidar data using an AI approach.
Before going to the core content, I would like to briefly introduce lidar to make sure we are on the same page. lidar is an acronym for Light Detection And Ranging. lidar sensors use pulsed light to create range scans or 3D models of a scene. The data that lidar creates is called lidar point cloud or just point clouds.
This technology is commonly used in three popular applications-- for autonomous perception, for aerial imagery, and more recently, in robotics and augmented reality applications. There are two key advantages in using lidar sensors. lidars provide high accurate depth measurement, which enables high accuracy in perceiving the environment around it. Secondly, it offers high-density data, which allows us to apply algorithms to detect and segment objects in the data.
Due to these reasons, lidar are being widely adopted in aerial applications, like mapping and surveying of large areas like cities, forests, agricultural lands, et cetera; inspection and monitoring of critical infrastructures; delivery and transport, such as package delivery; and in security and defense with UAVs for surveillance; and also, many more such applications.
Now, Rishu and I have worked with customers on these application areas and have helped them to adopt lidar processing workflows. We commonly come across three challenges faced by our customers when implementing these workflows. The first and most challenging aspect is that aerial point cloud processing is time and memory intensive. This is due to the large size of the lidar point cloud data.
The second is the distortion and tracking errors due to motion of the system. If you are new to aerial lidar processing, and you are not familiar with these terms, don't worry. We'll cover these later in the presentation. The third challenge is that when we have recorded point cloud data to train AI models, the process of labeling the data is manual and time consuming.
Now Rishu will go over and explain each challenge based on his field experience and will discuss how MATLAB and Simulink can help to address these challenges. Over to you, Rishu.
Thanks, Minhaj. So the first challenge in aerial point cloud processing is that it is highly time and memory intensive. This is because aerial lidar data is generally very large sized. Here we have a sample data from ground lidar as well as aerial lidars.
lidar point clouds captured using automotive lidars have small sizes, in the range of 100 to 200 meters, and therefore can fit into a memory for processing, whereas aerial lidar point clouds are collected over several miles and are large in size. This often leads to out of memory issues while we process the data. And since it is a single file, it is often difficult to apply parallel processing.
So the possible solution is to apply block processing. Minhaj, can you give us a brief idea about what is block processing, and how it can help in processing large point cloud data?
Sure, Rishu. So block processing of point clouds is the process of cropping large point cloud into smaller blocks, then process individual blocks, and finally stick them together or save it to memory. For this, we'll first find the block origin depending on the size of the whole point cloud and the size of blocks. We then store only block origin information instead of the whole point cloud. And this technique overcomes the out of memory issues we'll have.
In this workflow, the first step is to create blocks from the entire point cloud and store the block information. Next, using an adapter, we'll read individual blocks. An adapter is a MATLAB class object which reads and writes point cloud data serially or parallely based on our inputs. After this, we apply point cloud processing algorithms on the individual blocks separately.
Here, the processing algorithms is based on our application. It can be something simple like the de-noising or much complex like deep learning. Finally, we'll use the adapter to write the process point cloud and stitch them back together. We'll demonstrate this in the aerial lidar segmentation demo later in the session.
Thanks, Minhaj. It would be great to see a block processing in a demo. Looking forward. Now let's move on to the next challenge of distortion and tracking errors due to motion. Let's take a look at the first one, the distortion due to motion.
This is very common in high-speed applications. When the platform on which sliders are mounted is having high vibrations or is moving at a high velocity of 100 kilometers per hour or more, the point clouds captured by the sensor will start to get distorted. This leads to reduction of accuracy in the navigation and perception algorithms we would want to apply on the point cloud.
A possible solution is using other sensors such as IMUs along with lidar in the system, because IMUs can detect acceleration and vibrations of the system. And this can be used to analyze and correct the motion distortion of the collected point cloud.
The second one is the inaccuracy in tracking eco vehicles due to their motion. This is very important in UAV applications in urban environments, such as drone package delivery and air taxis. The safety of these operations is very much dependent on the ability to accurately track each of the non-eco vehicles. We can improve the accuracy of detection and tracking by adding radar and using lidar data with radar data.
Minhaj, can you show us how we can fuse lidar and radar data to improve the tracking accuracy?
Thanks, Rishu. To demonstrate this, I'll take an example from our documentation. In this example, we have a UAV with lidar and radar mounted on top of it. This UAV will track three other UAVs-- two quad rotors and an air taxi.
Let's quickly go through the steps. First, from lidar data we'll remove ground and buildings. And then we'll create bounding boxes around the UAVs to be tracked. After that, we'll use a JPDA tracker to track these bounding boxes.
On the other end, we'll track the radar data using an extended object tracker. We'll then implement track-based fusion to fuse the tracks from radar and lidar. Let's now compare the tracking algorithms from these three trackers.
From the output, we can determine that the fuse track result is better than individual lidar and radar tracking results. To confirm this, let's take Generalized Optimal Subpattern Assessment metric, or GOSPA metric. The GOSPA metric values evaluate the performance of a tracking system by providing a scalar cost. To put it simply, a lower value of the metric indicates better performance of the tracking algorithm.
We can see that we got a lower GOSPA value for a fuse track, which means it is the better tracking algorithm. To learn more about this, refer to our documentation. Now Rishu will talk about the next challenge.
Sure, Minhaj. So now let's talk about the third challenge of labeling point cloud. Labeling is very important to step in deep learning workflows or any supervised learning workflows. We need a good amount of labeled point cloud data sets to train a deep learning model for tasks such as object detection or semantic segmentation.
But the problem is, labeling is very repetitive, manual, and time consuming. And if you look at the image on the left, you will see two different point clouds which are captured at one after another, but there is a very little variation from frame to frame. And we will need to label both of them separately. Just imagine doing labeling for miles and miles of data.
Now, the second issue is noise. In the case of lidar, majority of the points in each scan are not objects of interest and need to be labeled. These are two ways engineers overcame this challenge. First, leveraging non AI-based techniques to automate part of the labeling process. second is to iteratively label a small set of data and train a model, and then use that model to help label the data and improve as we keep iterating. Both these options are in MATLAB. Minhaj, will you give us a brief idea about that?
Sure, Rishu. So MATLAB provides a Lidar Labeler App in Lidar Toolbox to interactively label point cloud data for semantic segmentation and object detection. Here I have loaded an aerial point cloud data into the app.
As a first step, I hide the ground from the point cloud for a better visualization of the object to be labeled. I then define the label name as trees, select voxels as label type, and color as green. After that, I can simply select the region where trees are to be labeled. I follow the same process to label buildings.
The third object I'll label are cars. For this, I use cuboid labeling. Generally, we don't use voxel labeling and cuboid labeling in the same session, because voxel labeling is for semantic segmentation, while cuboid labeling is done for object detection. But for now, I'm doing this to showcase the capabilities and features of the app.
As you can see, I'm using a projected view feature for a better visualization of objects of interest. Another useful feature in Lidar Labeler App is to automate labeling process with inbuilt or custom automation algorithm. Due to the time constraint, I'm not going to go through the steps. But you can refer to the Lidar Labeler Demo video we have in our video portal or in YouTube.
Once we are happy with the label point cloud, we can export it into a file or directly to MATLAB Workspace. So that's how you can use Lidar Labeler App to label your point cloud data.
Moving on to the next topic, I'll show you how you can segment aerial lidar data using AI. In this demo, we are applying a semantic segmentation network for a large point cloud using block processing. Semantic segmentation of point cloud is the process of classifying each point in the point cloud based on object that is present at that position.
We are using a state-of-the-art neural network PointNet++ which can segment the point cloud into different classes like vegetation, buildings, road, et cetera. First, I'll walk you through the steps involved in the segmentation. Then I'll show you how to implement the same using block processing workflow.
Here is a simplified view of any AI-driven systems. We have four main stages there. The first one is data preparation, and the second one is algorithms and modeling. Third one is simulation and testing. And finally, we have deployment.
In the data preparation step, we'll create the block, build the point cloud, and preprocess the point cloud to prepare it for applying AI model. In the second step of AI modeling, we'll define a PointNet++ network to segment the point cloud. In the third step, we'll see how we can test our model on a simulated data. And finally, we'll deploy the algorithm on a target hardware.
MATLAB gives you flexibility to read data from popular file formats such as LAS, LAZ, PCD, PCAP, et cetera. You can also stream live lidar data from Ouster and Velodyne sensors. In addition to this, you can also read point cloud from ROS bags.
Once we have loaded the point cloud data, we can use Lidar Viewer App to visualize, analyze, and edit point cloud data. We can import point cloud in different file formats, including LAS and LAZ. We can toggle between different types of visualization, like red to blue and parula. We can adjust the size of the points zoom in, zoom out, and pan. We can select different views, including custom views, and many more.
The key feature of the app allows us to apply basic preprocessing algorithms such as de-noising, downsampling, and ground removal on the point cloud interactively. Here I'm demonstrating how to remove ground and to crop point cloud data to an ROA.
Once we are done with the preprocessing, we can export the preprocessing functions, as well as the processed point cloud, into MATLAB Workspace. Now we have read the point cloud, preprocessed it, and visualized it using the Lidar Viewer App. You can also preprocess point cloud data using command line. You can find a variety of point cloud preprocessing functions in our documentation here.
In our workflow, we are implementing downsampling and normalizing function. We can use pcdownsample function on the point cloud without compromising its structures. We can normalize the point cloud based on the x, y, z limit and range with a few lines of code, as you can see here.
As the next step, we'll segment the point cloud. While our focus is on deep learning-based segmentation, I'll give you a brief overview on other segmentation techniques. We can segment the point cloud using three techniques. The first one is using clustering algorithms. The second is using features like normals and curvature values. And the third one is using semantic segmentation with a deep neural network.
In MATLAB, we can use pcsegdist function or segment lidar function for clustering. pcsegdist function segments point clouds into clusters based on the Euclidean distance between them. We'll give a minimum threshold value between the points from different clusters and then apply the function.
While pieces of this function can be used both on organized and unorganized point clouds, segment lidar function is designed to cluster organize point clouds, resulting in faster execution. We can also segment point clouds based on features like normals and curvatures of the points in the point cloud.
This uses the fact that the buildings are more planar than the vegetation. And vegetation points are more scattered, which then result in a higher change in value of curvature compared to the buildings. Using this difference, we'll segment the point load into buildings and vegetation.
The third technique is to use deep learning models. We'll be using state-of-the-art network PointNet++ to segment the point cloud. PointNet++ is a neural network used for semantic segmentation of an organized lidar point cloud. It classifies each point in the point cloud based on different classes like cars, trucks, drones, or vegetation.
The network contains an encoder with set abstraction modules and a decoder with feature propagation modules. The network tries to find local features and then group them into larger units to find higher-level features from lidar point clouds. You can refer to this MATLAB documentation page to learn more about PointNet++ network. If we zoom into our output, we can see that the output resembles the ground truth data.
Now let's see how we got this output using MATLAB's block processing workflow. As the first step, we'll define block size and create the block point cloud object and block point cloud data store. This data store will help us to manage the point clouds and avoid out of memory issues.
We'll then define a few parameters that we would need to apply the network, like class names and number of points in each block. After that, we load the pre-trained PointNet++ network. Now let's go on and read individual block from the data still using read function. And then we'll extract the labels from it.
We'll now process the point cloud block. As I mentioned before, we are mainly applying two preprocessing step here. The first one is downsampling and the second one is normalizing. After that, we'll apply the network on the block point cloud using semanticseg function. Finally, we'll assemble the point cloud block. And then we'll stitch the whole blocks together.
Here is our final output. You can see that we have segmented the input point cloud into different classes like buildings, trees, and ground. Now, if you want to test your network before applying it into a real-world system, you can use the simulation environment available in Simulink.
Here, I'm simulating a city scenario with a UAV flying based on a different trajectory. I have mounted a lidar sensor and defined a few sensor parameters based on my need. I can now run the model and collect the point cloud data, and then apply the AI model to test the model on the data.
Now that we have segmented the point cloud, let's move on to the final step of deployment. From MATLAB, you can generate C++ code as well as CUDA code for deep learning. We can use MATLAB Coder for C++ code generation.
We'll first specify MATLAB function name and then the input attributes. After that, we'll generate the code. The generated code provides a report which we can use to trace it back to the corresponding MATLAB code.
We can generate GPU code using GPU Coder. Refer to this example in Lidar Toolbox documentation, which explains the entire process of GPU code generation. You will need a CUDA-enabled GPU and a compatible driver for this.
We have demonstrated how to build a MATLAB system for semantic segmentation of aerial lidar data using AI. Before we wind up, I would like to mention a few very useful aerial lidar processing features. The first one is pc2dem function that can convert point cloud into Digital Elevation Models, or DEMs. DEMs are leveraged by many aerial applications, such as terrain modeling, terrain analysis, and flight simulation.
The second one is a workflow to create 3D maps. This demo uses simulated data from Unreal gaming environment. It uses Fast Point Feature Histogram descriptors, or FPFH descriptors, to register the point cloud. It also shows how to use pose graph optimization in case of drifts in the registration steps.
The third workflow shows forest masking and individual tree segmentation for forestry applications. This workflow will be useful in applications like estimating average height of trees, density of canopy, and amount of biomass. For more details about above workflows, visit our Lidar Toolbox product page.
In summary, we discussed common challenges faced by engineers and demonstrated how MATLAB and Simulink addressed these challenges. We showed how to read, write, and stream lidar point clouds, implement block processing on large point cloud, apply AI on lidar, automatic ground truth labeling, and deploy full applications, including pre- and post-processing, on target hardware.
We have now come to the end of this webinar. I hope this webinar was informative. And we would like to hear from you. We'll now open the floor for any questions. Thank you.