Label Ground Truth for Multiple Signals

Step 2 of 3 in Get Started with Ground Truth Labelling

After loading the video and lidar point cloud sequence signals into the Ground Truth Labeler app, as described in the Load Ground Truth Signals to Label procedure, create label definitions and label the signal frames. In this example, you label only a portion of the signals for illustrative purposes.

Create Label Definitions

Label definitions contain the information about the labels that you mark on the signals. You can create label definitions interactively within the app or programmatically by using a labelDefinitionCreatorMultisignal object. In this example, you create label definitions in the app.

Create ROI Label

An ROI label is a label that corresponds to a region of interest (ROI) in a signal frame. You can define these ROI label types.

Rectangle/Cuboid — Draw bounding box labels around objects, such as vehicles. In image signals, you draw labels of this type as 2-D rectangular bounding boxes. In point cloud signals, you draw labels of this type as 3-D cuboid bounding boxes.
Rotated rectangle — Draw 2-D bounding box labels around objects in an image that are rotated.
Projected cuboid — Draw 3-D bounding box labels around objects in an image, such as vehicles.
Line — Draw linear ROIs to label lines, such as lane boundaries.
Pixel label — Draw pixels to label various classes, such as road or sky, for semantic segmentation. For more information about pixel labeling, see Label Pixels for Semantic Segmentation
Polygon — Draw polygon labels around objects. You can label distinct instances of the same class. For more information on drawing polygon ROI labels for instance and semantic segmentation networks, see Label Objects Using Polygons

For more details about these ROI label definitions, see ROI Labels, Sublabels, and Attributes.

Create an ROI label definition for labeling cars in the signal frames.

On the ROI Labels pane in the left pane, click Label.
Create a Rectangle/Cuboid label named car.
From the Group list, select New Group and name the group Vehicles. Adding labels to groups is optional.
Click OK. The Vehicles group name appears on the ROI Labels tab with the label car under it.

The car label is drawn differently on each signal. On the video, car is drawn as a 2-D rectangular bounding box of type Rectangle. On the point cloud sequence, car is drawn as a 3-D cuboid bounding box of type Cuboid.

Create ROI Sublabel

A sublabel is a type of ROI label that corresponds to a parent ROI label. Each sublabel must belong to, or be a child of, a label definition that is in the ROI Labels tab. For example, in a driving scene, a vehicle label can have sublabels for headlights, license plates, or wheels. For more details about sublabels, see ROI Labels, Sublabels, and Attributes.

Create an ROI sublabel definition for labeling the brake lights of the labeled cars.

Select the parent label of the sublabel. On the ROI Labels tab in the left pane, click the car label to select it.
Click Sublabel.
Create a Rectangle sublabel named brakeLight. Cuboid sublabels are not supported, so this sublabel applies only for the video signal. Click OK.
The brakeLight sublabel appears in the ROI Labels tab under the car label. The sublabel and parent label have the same color.

Create ROI Attribute

An ROI attribute specifies additional information about an ROI label or sublabel. For example, in a driving scene, attributes can include the type or color of a vehicle. You can define ROI attributes of these types.

Numeric Value — Specify a numeric scalar attribute, such as the number of doors on a labeled vehicle.
String — Specify a string scalar attribute, such as the color of a vehicle.
Logical — Specify a logical true or false attribute, such as whether a vehicle is in motion.
List — Specify a drop-down list attribute of predefined strings, such as make or model of a vehicle.

For more details about these attribute types, see ROI Labels, Sublabels, and Attributes.

Create an attribute to describe whether a labeled brake light is on or off.

On the ROI Labels tab in the left pane, select the brakeLight sublabel and click Attribute.
In the Attribute Name box, type isOn. Set the attribute type to Logical. Leave Default Value set to Empty and optionally provide a description. Click OK. You can hover over the information icon that appears next to the attribute field to display the added description.
In the ROI Labels tab, expand the brakeLight sublabel definition. The Attribute box for this sublabel now contains the isOn attribute.

ROI Labels tab with an isOn attribute under the brakeLight label

Create Scene Label

A scene label defines additional information across all signals in a scene. Use scene labels to describe conditions, such as lighting and weather, or events, such as lane changes.

Create a scene label to apply to the signal frames.

In the left pane of the app, select the Scene Labels tab.
Click Define new scene label, and in the Label Name box, enter a scene label named daytime.
Change the color of the label definition to light blue to reflect the nature of the scene label. Under the Color parameter, click the color preview and select the standard light blue colors. Then, click OK to close the color selection window.
Leave the Group parameter set to the default of None and click OK. The Scene Labels pane shows the scene label definition.

Verify Label Definitions

Verify that your label definitions have this setup.

The ROI Labels tab contains a Vehicles group with a car label of type Rectangle/Cuboid.
The car label contains a sublabel named brakeLight.
The brakeLight sublabel contains an attribute named isOn.
The Scene Labels tab contains a light blue scene label named daytime.

To edit or delete a label definition, right-click that label definition and select the appropriate edit or delete option. To save these label definitions to a MAT-file for use in future labeling sessions, on the Label tab of the app toolstrip, first select Export. Then, in the Label Definitions section, select To File.

In future labeling sessions, if you need to reorder label definitions or move them to different groups, you can drag and drop them in the label definition panes.

Label Video Using Automation

Use the car label to label one of the cars in a portion of the video. To assist with the labeling process, use one of the built-in label automation algorithms.

Select the time range to label. Specify an interval from 8 to 10 seconds, during which the car in front is close to the ego vehicle. In the text boxes below the video, enter these times in this order:
1. In the Current box, type 8.
2. In the Start Time box, type 8 so that the slider is at the start of the time range.
3. In the End Time box, type 10.
The range slider and text boxes are set to this 8–10 second interval. The red flags indicate the start and end of the interval.
The app displays signal frames from only this interval, and automation algorithms apply to only this interval. To expand the time range to fill the entire playback section, click Zoom In Time Range.
Select the label that you want to automate. In the ROI Labels tab, click the car label.
Select the automation algorithm to use. From the app toolstrip, select Select Algorithm > Temporal Interpolator. This algorithm estimates rectangle ROIs between image frames by interpolating the ROI locations across the time range.
Select the signal that you want to automate. From the app toolstrip, click Select Signals. Then, in the Select Signals window, select only the video signal and click OK. This algorithm supports labeling of only one signal at a time, and the point cloud signal uses a different automation algorithm.
Click Automate. The app prompts you to confirm that you want to label only a portion of the video. Click Yes. An automation session for the video opens. The right pane of the automation session displays the algorithm instructions.
At the start of the time range, click and drag to draw a car label around the car in the center of the frame. For this algorithm, you can draw only one label per frame. Labeling the other car would require a separate automation session.

By default, the car label appears only when you move your pointer over it. To always display labels, on the app toolstrip, set Show ROI Labels to Always.
Drag the slider to the last frame and draw a car label around the same car in this frame. Optionally, to improve automation results, label the car in intermediate frames.
Click Run. The automation algorithm applies the car label to the intermediate frames. Drag the slider to view the results. If necessary, manually adjust the labels to improve their accuracy.
When you are satisfied with the results, click Accept to close the session and apply the labels to this portion of the video.

Label Point Cloud Sequence Using Automation

Use the same car label definition from the previous procedure to label a car in the point cloud sequence. To assist with the labeling process, use a built-in label automation algorithm designed for point cloud labeling. In this example, you label the ego vehicle, which is easier to see in the lidar point cloud sequence than the front car.

At the bottom of the app, verify that the time range is still set to 8 to 10 seconds.
In the labeling window, click the point cloud sequence to select it.
In the ROI Labels tab, click the car label definition.
On the Label tab of the app toolstrip, select Select Algorithm > Point Cloud Temporal Interpolator. This algorithm estimates cuboid ROIs between point cloud frames by interpolating the ROI locations across the time range.
Click Select Signals, select only the point cloud signal, and click OK.
Click Automate. The app prompts you to confirm that you want to label only a portion of the point cloud sequence. Click Yes. An automation session for the point cloud sequence opens. The right pane of the automation session displays the algorithm instructions.
At the start of the time range, draw a car label around the ego vehicle.
1. Zoom in on the car, using either the scroll wheel or the Zoom In button at the top-right corner of the frame. You can also use the Pan button to center the car in the frame.
2. On the ROI Labels tab in the left pane, click the car label. Drag the gray preview cuboid until it highlights the ego vehicle.
3. Click the signal frame to create the label. The label snaps to the highlighted portion of the point cloud.
4. Adjust the cuboid label until it fully encloses the car. To resize the cuboid, click and drag one of the cuboid faces. To move the cuboid, hold Shift and click and drag one of the cuboid faces.
5. Use projected view to adjust the cuboid label in top-view, side-view and front-view simultaneously. Under Lidar tab in the app toolstrip, select the Projected View option from the Camera View section, to enable this view.
  
  For additional tips and techniques for labeling point clouds, see Label Lidar Point Clouds for Object Detection.
Click the cuboid and press Ctrl+C to copy it. Then, drag the slider to the last frame and press Ctrl+V to paste the cuboid into the new frame at the same position. Optionally, to improve automation results, manually adjust the position of the copied label.
Click Run. The automation algorithm applies the car label to the intermediate frames. Drag the slider to view the results. If necessary, manually adjust the labels to improve their accuracy.
When you are satisfied with the results, click Accept to close the session and apply the labels to this portion of the point cloud sequence.

Label with Sublabels and Attributes Manually

Manually label one frame of the video with the brakeLight sublabel and its isOn attribute. Lidar point cloud signals do not support sublabels and attributes, so you cannot label the point cloud sequence.

At the bottom of the app, verify that the time range is still set to 8 to 10 seconds. If necessary, drag the slider to the first frame of the time range.
In the ROI Labels tab, click the brakeLight sublabel definition to select it.
Hide the point cloud sequence. On the Label tab of the app toolstrip, under Show/Hide Signals, clear the check mark for the lidar point cloud sequence. Hiding a signal only hides the display. The app maintains the labels for hidden signals, and you can still export them.
Expand the video signal to fill the entire labeling window.
In the video frame, select the drawn car label. The label turns yellow. You must select the car label (parent ROI) before you can add a sublabel to it.
Draw brakeLight sublabels for the car. Optionally, set Show ROI Labels to Always so that you can confirm the association between the car label and its sublabels.
On the video frame, select one of the brakeLight sublabels. Then, on the Attributes and Sublabels pane, set the isOn attribute to True. Repeat this step for the other sublabel.

For more details about working with sublabels and attributes, see Use Sublabels and Attributes to Label Ground Truth Data.

Label Scene Manually

Apply the daytime scene label to the entire scene.

Expand the time range back to the entire duration of all signals. If you zoomed in on the time range, first click Zoom Out Time Interval. Then, drag the red flags to the start and end of the range slider.
In the left pane of the app, select the Scene Labels tab.
Select the daytime scene label definition.
Above the label definition, click Time Interval. Then, click Add Label. A check mark appears for the daytime scene label indicating that the label now applies to all frames in the time range.

View Label Summary

With all labels, sublabels, and attributes applied to at least one frame of a signal, you can now optionally view a visual summary of the ground truth labels. On the app toolstrip, click View Label Summary. For more details, see View Summary of Ground Truth Labels.

Save App Session

On the app toolstrip, select Save Session and save a MAT-file of the app session. The saved session includes the data source, label definitions, and labeled ground truth. It also includes your session preferences, such as the layout of the app.

You can now either close the app session or continue to the Export and Explore Ground Truth Labels for Multiple Signals step, where you export the labels.