Main Content

Automate Testing for Lane Marker Detector

This example uses:

Open Script

This example shows how to automate testing of a lane marker detector algorithm and the generated code by using Simulink Test™. In this example, you will:

  • Assess the behavior of a lane marker detector algorithm on different test scenarios with different test requirements.

  • Automate testing of the lane marker detector algorithm and the generated code for the algorithm.

This example uses the lane marker detector model presented in Design Lane Marker Detector Using Unreal Engine Simulation Environment and Generate Code for Lane Marker Detector examples.

Introduction

A lane marker detector is a fundamental perception component of an automated driving application. The detector analyzes images of roads captured using a monocular camera sensor and returns information about the curvature and marking type of each lane. You can design and simulate a lane marker detector using MATLAB® or Simulink® and then, assess its accuracy using a known ground truth. You can configure the simulations to use test scenarios that are based on system requirements. You can integrate the detector to an external software environment and deploy to a vehicle through C++ code generation. Code generation and verification of the Simulink model ensures functional equivalence between simulation and real-time implementation. Automatically running these simulations enables regression testing to verify system-level functionality.

For information about how to design a lane marker detector and generate a model for C++ code generation, see Design Lane Marker Detector Using Unreal Engine Simulation Environment and Generate Code for Lane Marker Detector, respectively. This example shows how to automate testing the lane marker detector and the code generation models against multiple scenarios using Simulink Test™. The scenarios are based on system-level requirements. The rest of the example explains the steps involved in automated testing of the lane marker detector model.

  1. Review requirements: Explore the test scenarios and review the requirements that describe the test conditions.

  2. Review the test bench model: Review the lane marker detector test bench model that contains metric assessments. These metric assessments integrate the test bench model with Simulink Test™ for the automated testing.

  3. Disable runtime visualizations: Disable runtime visualizations to reduce execution time for the automated testing.

  4. Automate testing: Configure the test manager to simulate each test scenario, assess success criteria, and report the results. You can explore the results dynamically using the test manager and export to a PDF for external reviews.

  5. Automate testing with generated code: Configure the lane detection, sensor fusion, decision logic, and controls components to generate C++ code. Run automated testing on the generated code and verify behavior.

  6. Automate testing in parallel: Reduce overall execution time for the tests by using parallel computing on a multi-core computer.

In this example, you enable system-level simulation through integration with the Unreal Engine from Epic Games®.

if ~ispc
    error(['This example is supported only on Microsoft.', char(174), ...
        ' Windows', char(174), '.'])
end

To ensure reproducibility of the simulation results, set the random seed.

rng(0);

Review Requirements

This example contains six test scenarios for evaluating the model. To define the high-level testing requirements for each scenario, use Requirements Toolbox™.

To explore the test bench model, load the lane detector project.

openProject("LaneDetector");

Open the test requirements file.

open('LaneMarkerDetectorTestRequirements.slreqx')

You can also open the file using the Requirements tab of the Requirements Manager app in Simulink.

The panel displays information about the test scenarios and the test requirements in textual and graphical format. To test lane detection on scenarios with only ego vehicle, use scenario files with the prefix scenario_LF_. To test lane detection on scenarios with other vehicles on the road, use scenario files with the prefix scenario_LD_.

  1. scenario_LF_01_Straight_RightLane — Straight road scenario with ego vehicle in right lane.

  2. scenario_LF_02_Straight_LeftLane — Straight road scenario with ego vehicle in left lane.

  3. scenario_LF_03_Curve_LeftLane — Curved road scenario with ego vehicle in left lane.

  4. scenario_LF_04_Curve_RightLane — Curved road scenario with ego vehicle in right lane.

  5. scenario_LD_01_Curve_ThreeVehicles — Curved road scenario with a passing vehicle in ego lane.

  6. scenario_LD_02_Curve_SixVehicles — Curved road scenario with changing lead vehicles in ego lane.

These test scenarios have same names as the scenarios used in the LaneMarkerDetectorTestBench model.

Review Test Bench Model

This example reuses the LaneMarkerDetectorTestBench model from the Generate Code for Lane Marker Detector example.

Open the test bench model.

open_system("LaneMarkerDetectorTestBench");

This test bench model has Simulation 3D Scenario, Lane Marker Detector, and Metrics Assessment subsystems.

To configure the test bench model, use the helperSLLaneMarkerDetectorSetup script. Specify a test scenario as input to the setup script by using the scenarioName input argument. The value for scenarioName must be any one of the scenario names specified in the test requirements.

Run the setup script.

scenarioName = "scenario_LD_02_Curve_SixVehicles";
helperSLLaneMarkerDetectorSetup(scenarioName);

You can now simulate the model and visualize the results. For more details on the simulation and analysis of the simulation results, see the Design Lane Marker Detector Using Unreal Engine Simulation Environment and Generate Code for Lane Marker Detector example.

This example focuses on automating the simulation runs to test the lane marker detector on different driving scenarios by using Simulink Test. The Metrics Assessment subsystem integrates the system-level metric evaluations with Simulink Test by using the Check Static Range (Simulink) block.

Open the Metrics Assessment subsystem.

open_system("LaneMarkerDetectorTestBench/Metrics Assessment");

The Metric Assessment subsystem outputs:

  1. Left lane metrics: An array that contains the number of matches (true positives), misses (false negatives), and mismatches (false positives) computed from the detection results for left lane.

  2. Left lane status: A logical scalar returned as true (1) or false (0). The value is true for matches in the left lane and false for misses and false positives in the left lane.

  3. Left lane distance: A scalar specifying the average value of distances between detected left lane marking and the corresponding ground truth.

  4. Right lane metrics: An array that contains the number of matches (true positives), misses (false negatives), and false positives computed from the detection results for right lane.

  5. Right lane status: Returned as true or false. The value is true for matches in the right lane and false for misses and false positives in the right lane.

  6. Right lane distance: A scalar specifying the average value of distances between detected right lane boundary points and the ground truth for the right lane.

The model logs the output results from Metric Assessment subsystem to the base workspace variable logsout. You can verify the performance of the lane marker detector algorithm by validating and plotting the metrics computed separately for the left lane and the right lane during simulation. You can also verify the performance of lane marker detector algorithm post simulation using precision and sensitivity metrics.

Disable Runtime Visualizations

Disable runtime visualizations for the Lane Marker Detector subsystem.

load_system('LaneMarkerDetector');
blk = 'LaneMarkerDetector/Lane Marker Detector';
set_param(blk,'EnableDisplays','off');

Configure the Simulation 3D Scene Configuration block to run the Unreal Engine in headless mode, where the 3D simulation window is disabled.

blk = ['LaneMarkerDetectorTestBench/Simulation 3D Scenario/', ...
      'Simulation 3D Scene Configuration'];
set_param(blk,'EnableWindow','off');

Automate Testing

Open the LaneMarkerDetectorTestAssessments.mldatx test file in the Test Manager. The Test Manager is configured to automate the testing of lane marker detector algorithm.

sltestmgr;
testFile = sltest.testmanager.load('LaneMarkerDetectorTestAssessments.mldatx');

The test cases in the Test Manager are linked to the test requirements in the Requirements Editor. Each test case uses the POST-LOAD callback to run the setup script with appropriate inputs and to configure the output video file name. After the simulation of the test case, it invokes helperGenerateReportForLaneMarkerDetector from the CLEAN-UP callback to generate the plots for lane boundary parameters and to generate the videos with overlaid lane detections. It also invokes CUSTOM CRITERIA to plot detection results from Metric Assessment subsystem and to verify the precision and sensitivity metrics using helperVerifyPrecisionAndSensitivity function. For more information about these plots, see Generate Code for Lane Marker Detector example.

Run and explore results for a single test scenario:

Test the system-level model on scenario_LD_02_Curve_SixVehicles scenario.

testSuite = getTestSuiteByName(testFile,'Test Scenarios');
testCase = getTestCaseByName(testSuite,'scenario_LD_02_Curve_SixVehicles');
resultObj = run(testCase);

Generate test reports obtained after simulation.

sltest.testmanager.report(resultObj,'Report.pdf',...,
    'Title','Lane Marker Detector',...
    'IncludeMATLABFigures',true,...
    'IncludeErrorMessages',true,...
    'IncludeTestResults',0,'LaunchReport',true);

Examine the Report.pdf. The Test environment section shows the platform on which the test is run and the MATLAB® version used for testing. The Summary section shows the outcome of the test and duration of the simulation in seconds. The Results section shows pass or fail results based on the assessment criteria and displays the plots logged from the helperGenerateReportForLaneMarkerDetector function.

Run and explore results for all test scenarios:

You can simulate the system to run all the tests by using sltest.testmanager.run command. Alternatively, you can click Play in the Test Manager app.

After completion of the test simulations, the results for all the tests can be viewed in the Results and Artifacts tab of the Test Manager. For each test case, the Check Static Range (Simulink) blocks in the model are associated with the Test Manager. You can visualize the overall pass or fail results.

You can find the generated report in current working directory. This report contains a detailed summary of pass or fail statuses and the plots for each test case.

Verify test status in Requirements Editor:

Open the Requirements Editor and select Display. Then, select Verification Status to see a verification status summary for each requirement. Green and red bars indicate the pass and fail status of simulation results for each test.

Automate Testing with Generated Code

The LaneMarkerDetectorTestBench model enables testing of Lane Marker Detector component. To perform regression testing of these components use software-in-the-loop (SIL) verification. If you have Embedded Coder™ Simulink Coder™ license, you can generate code for the lane marker detector component. You can verify if the generated code produce results that match the test requirements.

Set the simulation mode for Lane Marker Detector to software-in-the-loop.

model = 'LaneMarkerDetectorTestBench/Lane Marker Detector';
set_param(model,'SimulationMode','Software-in-the-loop');

Simulate the system for all the test scenarios and generate the test report by using the MATLAB command, run sltest.testmanager.run. Use the generated report to review the plots and test results.

If you have a Simulink Coverage™ license, you can get the code coverage analysis for the generated code to measure the testing completeness. You can use missing coverage data to find gaps in testing, missing requirements, or unintended functionality. You can visualize the coverage results for individual test cases and also aggregated coverage results as well.

You can click on LaneMarkerDetector link in test manager to view the detailed report of coverage results.

Automate Testing in Parallel

If you have a Parallel Computing Toolbox™ license, you can configure the test manager to execute tests in parallel using a parallel pool. To run the tests in parallel, disable the runtime visualizations and save the models using save_system('LaneMarkerDetector') and save_system('LaneMarkerDetectorTestBench'). Test Manager uses the default Parallel Computing Toolbox cluster and executes tests only on the local machine. Running tests in parallel speeds up execution and decreases the amount of time required for testing. For more information on how to configure tests in parallel using the Test Manager, see Run Tests Using Parallel Execution (Simulink Test).

See Also

Blocks

Related Topics