evaluateSemanticSegmentation

Evaluate semantic segmentation data set against ground truth

Syntax

ssm = evaluateSemanticSegmentation(dsResults,dsTruth)

ssm = evaluateSemanticSegmentation(imageSetConfusion,classNames)

[ssm,blockMetrics] = evaluateSemanticSegmentation(blockSetConfusion,classNames)

[___] = evaluateSemanticSegmentation(___,Name,Value)

Description

ssm = evaluateSemanticSegmentation(dsResults,dsTruth) computes various metrics to evaluate the quality of the semantic segmentation results, dsResults, against the ground truth segmentation, dsTruth.

ssm = evaluateSemanticSegmentation(imageSetConfusion,classNames) computes various metrics to evaluate the quality of the semantic segmentation results from confusion matrices, imageSetConfusion, with segmentation classes classNames.

[ssm,blockMetrics] = evaluateSemanticSegmentation(blockSetConfusion,classNames) computes various metrics to evaluate the quality of the block-based semantic segmentation results from confusion matrices, blockSetConfusion with classes classNames.

[___] = evaluateSemanticSegmentation(___,Name,Value) computes semantic segmentation metrics using one or more Name,Value pair arguments to control the evaluation.

Examples

Evaluate Semantic Segmentation Results

The triangleImages data set has 100 test images with ground truth labels. Define the location of the data set.

dataSetDir = fullfile(toolboxdir("vision"),"visiondata","triangleImages");

Define the location of the test images and ground truth labels.

testImagesDir = fullfile(dataSetDir,"testImages");
testLabelsDir = fullfile(dataSetDir,"testLabels");

Create an imageDatastore holding the test images.

imds = imageDatastore(testImagesDir);

Define the class names and their associated label IDs.

classNames = ["triangle" "background"];
labelIDs   = [255 0];

Create a pixelLabelDatastore holding the ground truth pixel labels for the test images.

pxdsTruth = pixelLabelDatastore(testLabelsDir,classNames,labelIDs);

Load a semantic segmentation network that has been trained on the training images of triangleImages.

net = load("triangleSegmentationNetwork");
net = net.net;

Run the network on the test images. Predicted labels are written to disk in a temporary directory and returned as a pixelLabelDatastore.

pxdsResults = semanticseg(imds,net,Classes=classNames,WriteLocation=tempdir);

Running semantic segmentation network
-------------------------------------
* Processed 100 images.

Evaluate the prediction results against the ground truth.

metrics = evaluateSemanticSegmentation(pxdsResults,pxdsTruth);

Evaluating semantic segmentation results
----------------------------------------
* Selected metrics: global accuracy, class accuracy, IoU, weighted IoU, BF score.
* Processed 100 images.
* Finalizing... Done.
* Data set metrics:

    GlobalAccuracy    MeanAccuracy    MeanIoU    WeightedIoU    MeanBFScore
    ______________    ____________    _______    ___________    ___________

       0.99074          0.99183       0.91118      0.98299        0.80563  

Display the properties of the semanticSegmentationMetrics object.

metrics

metrics = 
  semanticSegmentationMetrics with properties:

              ConfusionMatrix: [2×2 table]
    NormalizedConfusionMatrix: [2×2 table]
               DataSetMetrics: [1×5 table]
                 ClassMetrics: [2×3 table]
                 ImageMetrics: [100×5 table]

Display the classification accuracy, the intersection over union, and the boundary F-1 score for each class. These values are stored in the ClassMetrics property.

metrics.ClassMetrics

ans=2×3 table
                  Accuracy      IoU      MeanBFScore
                  ________    _______    ___________

    triangle      0.99302     0.83206      0.67208  
    background    0.99063      0.9903      0.93918  

Display the normalized confusion matrix that is stored in the NormalizedConfusionMatrix property.

metrics.ConfusionMatrix

ans=2×2 table
                  triangle    background
                  ________    __________

    triangle        4697           33   
    background       915        96755   

Input Arguments

dsResults — Predicted pixel labels

datastore | cell array of datastore objects

Predicted pixel labels resulting from semantic segmentation, specified as a datastore or a cell array of datastore objects. dsResults can be any datastore that returns categorical images, such as PixelLabelDatastore. The read(dsResults) must return a categorical array, a cell array, or a table. If the read function returns a multicolumn cell array or table, the second column must contain categorical arrays.

dsTruth — Ground truth pixel labels

datastore | cell array of datastore objects

Ground truth pixel labels, specified as a datastore or a cell array of datastore objects. dsTruth can be any datastore that returns categorical images, such as PixelLabelDatastore. Using read(dsTruth) must return a categorical array, a cell array, or a table. If the read function returns a multicolumn cell array or table, the second column must contain categorical arrays.

imageSetConfusion — Confusion matrices for segmented images

table | cell array

Confusion matrix for the classes in the segmented images, specified as one of the following, where F is the number of images in the data set.

• Table with F rows and one variable with the name ConfusionMatrix. Each row in the table contains a cell array with the confusion matrix for the corresponding image.

• F-by-one cell array. Each element of the cell array contains the confusion matrix for the corresponding image.

blockSetConfusion — Confusion matrices for segmented blocks

table

Confusion matrices for segmented blocks, specified as a table with B rows and three columns, where B is the total number of blocks in all images in the data set. The three columns are the variables ImageNumber, ConfusionMatrix, and BlockInfo. You can obtain a table of the correct format by using the segmentationConfusionMatrix function within a call to the block-based apply function. For an example, see Calculate Segmentation Metrics in Block-Based Workflow.

classNames — Class names

array of strings | cell array of character vectors

Class names, specified as an array of strings or a cell array of character vectors.

Example: ["sky" "grass" "building" "sidewalk"]

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: metrics = evaluateSemanticSegmentation(pxdsResults,pxdsTruth,Metrics="bfscore") computes only the mean BF score of each class, each image, and the entire data set.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: metrics = evaluateSemanticSegmentation(pxdsResults,pxdsTruth,"Metrics","bfscore") computes only the mean BF score of each class, each image, and the entire data set.

Metrics — Segmentation metrics

"all" (default) | vector of strings

Segmentation metrics in semanticSegmentationMetrics to compute, specified as a vector of strings. This argument specifies which variables in the DataSetMetrics, ClassMetrics, and ImageMetrics tables to compute. ConfusionMatrix and NormalizedConfusionMatrix are computed regardless of the value of Metrics.

Value	Description	Aggregate Data Set Metric	Image Metric	Class Metric
"all"	Evaluate all semantic segmentation metrics. The function excludes MeanBFScore from the semantic segmentation metrics when you specify a confusion matrix (imageSetConfusion or blockSetConfusion) as input to the function.	All aggregate data set metrics	All image metrics	All class metrics
"accuracy"	Accuracy indicates the percentage of correctly identified pixels for each class. Use the accuracy metric if you want to know how well each class correctly identifies pixels. For each class, Accuracy is the ratio of correctly classified pixels to the total number of pixels in that class, according to the ground truth. In other words, Accuracy score = TP / (TP + FN) TP is the number of true positives and FN is the number of false negatives. For the aggregate data set, MeanAccuracy is the average Accuracy of all classes in all images. For each image, MeanAccuracy is the average Accuracy of all classes in that particular image. The class accuracy is a simple metric analogous to global accuracy, but it can be misleading. For example, labeling all pixels "car" gives a perfect score for the "car" class (although not for the other classes). Use class accuracy in conjunction with IoU for a more complete evaluation of segmentation results.	MeanAccuracy	MeanAccuracy	Accuracy
"bfscore"	The boundary F1 (BF) contour matching score indicates how well the predicted boundary of each class aligns with the true boundary. Use the BF score if you want a metric that tends to correlate better with human qualitative assessment than the IoU metric. For each class, MeanBFScore is the average BF score of that class over all images. For each image, MeanBFScore is the average BF score of all classes in that particular image. For the aggregate data set, MeanBFScore is the average BF score of all classes in all images. For more information, see bfscore. This metric is not available when you specify a confusion matrix (imageSetConfusion or blockSetConfusion) as input to the function.	MeanBFScore	MeanBFScore	MeanBFScore
"global-accuracy"	GlobalAccuracy is the ratio of correctly classified pixels, regardless of class, to the total number of pixels. Use the global accuracy metric if you want a quick and computationally inexpensive estimate of the percentage of correctly classified pixels.	GlobalAccuracy	GlobalAccuracy	none
"iou"	Intersection over union (IoU), also known as the Jaccard similarity coefficient, is the most commonly used metric. Use the IoU metric if you want a statistical accuracy measurement that penalizes false positives. For each class, IoU is the ratio of correctly classified pixels to the total number of ground truth and predicted pixels in that class. In other words, IoU score = TP / (TP + FP + FN) The image describes the true positives (TP), false positives (FP), and false negatives (FN). For each image, MeanIoU is the average IoU score of all classes in that particular image. For the aggregate data set, MeanIoU is the average IoU score of all classes in all images. For more information, see jaccard.	MeanIoU	MeanIoU	IoU
"weighted-iou"	Average IoU of each class, weighted by the number of pixels in that class. Use this metric if images have disproportionally sized classes, to reduce the impact of errors in the small classes on the aggregate quality score.	WeightedIoU	WeightedIoU	none

Example: ["global-accuracy","iou"] calculates the global accuracy and IoU metrics across the data set, images, and classes.

Data Types: string

Verbose — Flag to display evaluation progress

1 (default) | 0

Flag to display evaluation progress information in the command window, specified as 1 (true) or 0 (false).

The displayed information includes a progress bar, elapsed time, estimated time remaining, and data set metrics. When Verbose is 0 (false), the evaluateSemanticSegmentation function calculates segmentation metrics without displaying progress information.

Data Types: logical

Output Arguments

ssm — Semantic segmentation metrics

semanticSegmentationMetrics object

Semantic segmentation metrics, returned as a semanticSegmentationMetrics object.

blockMetrics — Block-based semantic segmentation metrics

F-by-one cell array

Block-based semantic segmentation metrics, returned as an F-by-one cell array, where F is the number of images in the data set. Each element in the cell array contains information about all of the metrics calculated for all blocks in the corresponding image, formatted as a table.

Each table has K(f) rows, where K(f) is the number of blocks in the fth image in the data set. The table has up to five variables:

• The table always includes the BlockInfo variable. This table data in this variable are structs that provide spatial information about the block. The four fields of the struct are BlockStartWorld, BlockEndWorld, DataStartWorld, and DataEndWorld. For more information about these fields, see the IncludeBlockInfo name-value pair argument of the apply function.

• The table includes the metrics in the DataSetMetrics property of the ssm output argument. By default, the metrics are GlobalAccuracy, MeanAccuracy, MeanIoU, and WeightedIoU. However, if you create the ssm and specify a subset of the metrics to calculate by using the Metrics name-value argument, then the table includes only the specified metrics.

Tips

• A value of NaN in the dataset, class, or image metrics, indicates that one or more classes were missing during the computation of the metrics when using the evaluateSemanticSegmentation function. In this case, the software was unable to accurately compute the metrics.

  The missing classes can be found by looking at the ClassMetrics property, which provides the metrics for each class. To more accurately evaluate your network, augment your ground truth with more data that includes the missing classes.

References

[1] Csurka, G., D. Larlus, and F. Perronnin. "What is a good evaluation measure for semantic segmentation?" Proceedings of the British Machine Vision Conference, 2013, pp. 32.1–32.11.

Extended Capabilities

Automatic Parallel Support

Accelerate code by automatically running computation in parallel using Parallel Computing Toolbox™.

• To run in parallel, set 'UseParallel' to true or enable this by default using the Computer Vision Toolbox™ preferences.

  For more information, see Parallel Computing Toolbox Support.

• Parallel processing is only supported when the input datastores are pixelLabelDatastore objects.

Version History

Introduced in R2017b