Run Same Test with Different Inputs in Polyspace Platform User Interface

If you want to write multiple C/C++ tests that differ only in the input values or expected values of assessments, you do not have to duplicate the same test several times. Instead, you can write a parameterized test in the Polyspace Platform user interface. In a parameterized test, you specify parameters for test inputs instead of constants, and then iterate the parameter through a test data array.

Example Files

This tutorial uses the files in the folder polyspaceroot\polyspace\examples\doc_pstest\parametrized_tests_ui\src. Here, polyspaceroot is the Polyspace® installation folder, for instance, C:\Program Files\Polyspace\R2026a. To continue with this tutorial:

  1. Create a new Polyspace Platform project and add the folder to the project.

  2. Select Parse Code on the toolstrip to analyze the files in the folder.

Decide Which Inputs to Parameterize

Before writing a parameterized test, decide on a testing strategy and figure out which tests can be combined by writing a single parameterized test.

For instance, consider the function getDifferenceOfValues that calculates the magnitude of difference between the absolute values of its inputs:

#include <stdlib.h>

int32_t getDifferenceOfValues(int32_t arg1, int32_t arg2) {
    return abs(abs(arg1) - abs (arg2));
}
You might have a testing strategy where you fix the second argument to a value, for instance, the value 1, and iterate the first argument through a series of values:

  • -2 (Magnitude greater than 1, opposite sign)

  • -1 (Magnitude same as 1, opposite sign)

  • 0 (Magnitude less than 1)

  • 1 (Magnitude same as 1, same sign)

  • 2 (Magnitude same as 1, same sign)

Instead of writing five separate tests, you can write a single parameterized test to implement this testing strategy. Note that parameterizing inputs also requires your assessment values to be parameterized.

Parameterize Inputs and Assessments

Write a parameterized test to implement the test strategy outlined in the previous section:

  1. On the Projects pane, right-click getDifferenceOfValues and select Add Test Case.

  2. In the Inputs section of the test, right-click the Values cell for the input that you want to parameterize, for instance, arg1 in the preceding example, and select Add test parameter.

    Right-clicking entry in value column and select option to add test parameter.

  3. Enter a name for the parameter and the number of values the parameter can take.

    Entering a test parameter name and size

    You can see the parameter arg1_values appear in the Test Parameters section of the test.

  4. In the Test Parameters section, fill in the values that the parameter can take.

    Entering values of test input parameter

    Note that:

    • For aggregates such as arrays and structures, instead of entering individual member values in each row, you can enter their values all at once in a C-style initializer list. Right-click the row containing the aggregate and select Fill Using Initializer List. For more information, see Fill Aggregates Using C-Style Initializer Lists.

    • You can resize a test parameter after the initial parameter creation. To resize a target, right-click the root node corresponding to the parameter, for instance, arg1_values in the example, and select Resize Test Data.

  5. Enter the value 1 for the input arg2.

  6. Repeat the same steps for the assessment value and parameterize the assessment.

    1. In the Assessments section, right-click the Value cell and select Add test parameter.

    2. Enter a parameter name, for instance, return_values, and size.

      The size of an assessment parameter depends on the size of input parameters and the iteration strategy. If you parameterize only one input, the assessment parameter size is the same as the input parameter size.

    3. Fill in the assessment parameter values. You can calculate each assessment parameter value based on the corresponding input parameter value.

      For instance, in the preceding example:

      • The input arg1 is a parameter that goes through the array {-2,-1,0,1,2}.

      • The input arg2 takes the fixed value 1.

      Therefore, the return value of the function getDifferenceOfValues must be a parameter that goes through the array {1,0,1,0,1}.

      Entering values of test assessment parameter

Decide Parameterization Strategy

After you parameterize inputs and assessments appropriately, you can decide how you want parameter values to appear in your tests. You can then build and run the parameterized test, just like any other test. See Build and Run Tests in Polyspace Platform User Interface.

You can specify the following parametrization strategies.

Decide Between Sequential and Exhaustive Combinations

If you parameterize multiple inputs or assessments, specify one of these parameter iteration strategies:

  • Sequential – If you use this strategy, the parameter values are applied in sequence. The first test iteration uses the first values of all parameters, the second iteration uses the second values of all parameters, and so on.

  • Exhaustive – If you use this strategy, all parameter value combinations are exhaustively tested. That is, the test iterations are such that each value of a test parameter is combined with each value of other test parameters.

    Note that the exhaustive parameter combination strategy does not distinguish between parameters for inputs and assessments. Therefore, this strategy is useful only when you do not need to parameterize the expected values in the test assessments. For instance:

    • You might be testing a function that returns an error code in case of an error. In all your test iterations, you can test that an error did not occur by assessing for the single return value that represents the absence of an error.

    • You might be using a comparator other than == to test if a function returns values within a given range. Your assessments can use the >, >=, <, or <= comparators with a single expected value representing a boundary of the range.

In this example, you want each value of the test assessment parameter to be paired with the corresponding value of the test input parameter. Select the Sequential strategy to enable this pairing.

Decide Between Preload and Run-time Load of Parameter Values

Instead of embedding all parameter values in the generated test code, you can pass the parameter values for each iteration at runtime. To enable run-time load of parameter values, select the option Load Parameters at Runtime.

This option helps reduce the test executable size for parameterized tests in case you want to run the tests on embedded targets with memory restrictions. Since the data is passed at runtime, selecting this option can result in slightly increased test execution times.

Note that parameters of all data types cannot be loaded at runtime. If you choose to load parameters at runtime, the inputs that you parameterize can have only these data types or typedef-s of these data types:

  • Integer types (char, short, int, long, and long long)

    Note that most integer types such as size_t, wchar_t and fixed-width types such as int32_t are defined as typedef-s to these types.

  • Unsigned integer types

  • Floating-point types (float and double)

  • Boolean types

  • Aggregate data types such as structures and arrays, where each individual member has a supported data type

When you select this option, if any of your parameters have an unsupported type, you see an error when you validate or build the test. Check the Logs pane for more details of the error.

In this example, all inputs have data types that are supported for runtime loading of parameters. You can leave the option unselected, or select the option to see its effect on test execution.

Review Test Results

Build and run the tests. After the test execution is completed, you see a Tests node on the Results pane. To review the test results:

  1. Right-click the Tests node and select Open Dashboard. You see an overview of the results on a dashboard.

    Dashboard showing how many tests passed

    This dashboard shows that the parameterized test you wrote translated to five test iterations. All iterations passed.

  2. Click the Review button on the Dashboard tab of the Polyspace Platform toolstrip.

    You see all test iterations listed on the Results List pane. The Result column shows that all tests passed. You can see the same information on the Result Details pane when you select the test case node (for failed tests, this pane also shows expected and actual values).

    Results list showing iterations of a parameterized test

    Since the parameterized test has two parameters, each iteration is indexed by a pair of numbers.

    • The iteration titled New Test Case (1,1) indicates that the first parameter arg1_values takes the first value, -2, from the list of parameter values in the Test Parameters section, and the second parameter return_values also takes the first value, 1.

    • The iteration titled New Test Case (2,2) indicates that the first parameter arg1_values takes the second value, -1, from the list of parameter values in the Test Parameters section, and the second parameter return_values also takes the second value, 0.

    And so on.

    For failing test iterations, you can click on the row displaying the iteration and see the expected and actual value for each failing assessment on the Result Details pane. To see which parameter values were used in an iteration, you can click on any row on the Results List pane. This action opens a read-only form of the test case definition on the Test Definition pane. Using the indices on the results list together with the test definition, you can determine the test parameter values used in an iteration.

For more information, see Review C/C++ Test Execution Results.

Limitations

You cannot parameterize inputs or assessments when the input variable, or the variable being assessed, has one of the following types:

  • A union type or an aggregate type containing a union as a child

  • A pointer type

For these cases, do one of the following:

  • Write a test for a single set of inputs and assessments. To create a variant of this test, right-click the test on the Projects pane and select Duplicate Test. In the duplicated test, modify inputs and assessments as needed.

  • For pointer inputs, you can also write a scripted test with a step body that has a call to the function under test. In the function call, for pointer inputs, pass the address of a variable and then outside the step body, parametrize this variable.

    For instance, to test a function func(int *), write a scripted test with the step body:

    ret = func(&var);
    Add var as a step input (and optionally assessment), and then create a test parameter for var. Likewise, add ret as a step observable and assessment, and create a second test parameter for ret. For more information on creating scripted tests, see Test C/C++ Functions by Using Scripts in Graphical Tests.

See Also

Topics