Fix Metric Threshold Violations in a Continuous Integration Systems Workflow
This example shows how to use the Metrics Dashboard with open-source tools GitLab™ and Jenkins™ to test and refine your model in a continuous integration systems workflow. Continuous integration is the practice of merging all developer working copies of project files to a shared mainline. This workflow saves time and improves quality by maintaining version control and automating and standardizing testing.
This example refers to a project that contains the shipped project matlab:sldemo_slproject_airframe and these files that you must provide:
A MATLAB script that specifies metric thresholds and customizes the Metrics Dashboard.
A MATLAB unit test that collects metric data and checks whether there are metric threshold violations.
This example uses the Jenkins continuous integration server to run the MATLAB unit test to determine if there are metric threshold violations. Jenkins archives test results for you to download and investigate locally. GitLab is an online Git repository manager that you can configure to work with Jenkins. This diagram shows how Simulink Check, GitLab, and Jenkins work together in a continuous integration workflow.
In addition to the files in the matlab:sldemo_slproject_airframe project, you must provide these additional files:
A MATLAB unit test that collects metric data for the project and checks that the model files contain no metric threshold violations. For more information on the MATLAB Unit tests, see Script-Based Unit Tests .
A MATLAB script that specifies metric thresholds and customizes the Metrics Dashboard. For more information on how to customize the Metrics Dashboard, see Customize Metrics Dashboard Layout and Functionality.
setup.mfile that activates the configuration XML files that define metric thresholds, sets custom metric families, and customizes the Metrics Dashboard layout. For this example, the
setup.mscript contains this code:On the Project tab, click Startup Shudown. For the Startup files field, specify the
function setup % refresh Model Advisor customizations Advisor.Manager.refresh_customizations(); % set metric configuration with thresholds configFile = fullfile(pwd, 'config', 'MyConfiguration.xml'); slmetric.config.setActiveConfiguration(configFile); uiconf = fullfile(pwd, 'config', 'MyDashboardConfiguration.xml'); slmetric.dashboard.setActiveConfiguration(uiconf); end
sl_customization.mfile that activates the Model Advisor configuration file to customize the Model Advisor checks. For more information on creating your own Model Advisor configuration, see Configure Compliance Metrics.
runscript that executes during a Jenkins build. For this example, this code is in the
% script executed during Jenkins build function run(IN_CI) if (IN_CI) jenkins_workspace = getenv('WORKSPACE'); cd(jenkins_workspace); end % open the sl project slproj = simulinkproject(pwd); % execute tests runUnitTest(); slproj.close(); if IN_CI exit end end
cleanup.mfile that resets the active metric configuration to the default configuration. For this example, this code is in the
cleanup.mfile script:On the Project tab, click Startup Shudown. For the Shutdown files field, specify the
function cleanup rmpath(fullfile(pwd, 'data')); Advisor.Manager.refresh_customizations(); % reset active metric configuration to default slmetric.config.setActiveConfiguration(''); slmetric.dashboard.setActiveConfiguration(''); end
.gitignorefile that verifies that derived artifacts are not checked into GitLab. This code is in the
work/** reports/** *.asv *.autosave
Create a GitLab project for source-controlling your Project. For more information, see https://docs.gitlab.com/ee/README.html.
Install the Git client.
Set up a branching workflow. With GitLab, from the main branch, create a temporary branch for implementing changes to the model files. Integration engineers can use Jenkins test results to decide whether to merge a temporary branch into the main branch. For more information, see
Under Settings > Repository, protect the main branch by enforcing the use of merge requests when developers want to merge their changes into the main branch.
Under Settings, on the Integrations page, add a webhook to the URL of your Jenkins project. This webhook triggers a build job on the Jenkins server.
Install GitLab and TAP plugins. The MATLAB unit test uses the TAP plugin to stream
results to a
.tap file. To enable communication of the test status
from MATLAB to the Jenkins job, Jenkins imports the
Create a Jenkins project. Specify these configurations:
In your Jenkins project, click Configure.
On the General tab, specify a project name.
On the Source Code Management tab, for the Repository URL field, specify the URL of your GitLab repository.
On the Build Triggers tab, select Build when a change is pushed to GitLab.
On the Build tab, execute MATLAB to call the
runscript opens the project and runs all unit tests. For the project in this example, the code is:
matlab -nodisplay -r... "cd /var/lib/jenkins/workspace/'18b Metrics CI Demo'; run(true)"
In the Post-build Actions tab, configure the TAP plugin to publish TAP results to Jenkins. In the Test Results field, specify
reports/*.tap. For Files to archive, specify
The TAP plugin shows details from the MATLAB Unit test in the extended results of the job. The Jenkins archiving infrastructure saves derived artifacts that are generated during a Jenkins build.
Continuous Integration Workflow
After setting up your project, Jenkins, and GitLab, follow the continuous integration workflow.
Phase 1: Feature Development
Create a local clone of the GitLab repository. See Retrieve Files from Git Repository.
In Simulink, navigate to the local GitLab repository.
Make any necessary changes to the project files.
Simulate the model and validate the output in the Simulation Data Inspector.
Run MATLAB unit tests. For more information, see
In GitLab, create a merge request. Select the feature branch as source branch and the target branch as main. Click Compare Branches and Continue.
If the feature is not fully implemented, mark the merge request as a work in progress by adding the letters WIP: at the beginning of the request. If the merge request is not marked WIP:, it immediately triggers a build after creation.
Click Submit Merge Request.
Phase 2: Qualification by Using Continuous Integration
If the letters WIP: are not at the beginning of the merge request, the push command triggers a Jenkins build. In the Jenkins Setup part of this example, you configured Jenkins to perform a build when you pushed changes to GitLab. To remove the letters, click Resolve WIP status.
Navigate to the Jenkins project. In Build History, you can see the build status.
Click the Build.
Click Tap Test Results.
For this example, the
MetricThresholdGateway.munit test did not pass for three metrics because these metrics did not meet the thresholds. To investigate this data, you must download the data locally.
Phase 3: Investigate Quality Issues Locally
Download the archived results to a local Git repository workspace.
Unzip the downloaded files. Copy the
work/folders to the respective folders in the local repository.
To explore the results, open the project and the Metrics Dashboard.
To resolve the test failures, make the necessary updates to the models. Push the changes to the feature branch in GitLab.
Integration engineers can use Jenkins test results to decide when it is acceptable to perform the merge of the temporary branch into the main branch.
- Collect Model Metric Data by Using the Metrics Dashboard
- Collect and Explore Metric Data by Using the Metrics Dashboard