anomalyDetectionMetrics

Anomaly detection metrics

Since R2022b

Description

An anomalyDetectionMetrics object encapsulates anomaly detection quality metrics for a set of images. Images of anomalies belong to the positive (true) class. Normal images belong to the negative (false) class.

Note

This functionality requires the Automated Visual Inspection Library for Computer Vision Toolbox™. You can install the Automated Visual Inspection Library for Computer Vision Toolbox from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons.

Creation

Create an anomalyDetectionMetrics object using the evaluateAnomalyDetection function.

Properties

ConfusionMatrix — Confusion matrix

table

This property is read-only.

Confusion matrix, specified as a 2-by-2 table. Each table element (i, j) is the count of images known to belong to class i but predicted to belong to class j.

NormalizedConfusionMatrix — Normalized confusion matrix

table

This property is read-only.

Normalized confusion matrix, specified as a 2-by-2 table. NormalizedConfusionMatrix represents a confusion matrix normalized by the number of images known to belong to each class. Each table element (i, j) is the count of images known to belong to class i but predicted to belong to class j, divided by the total number of images predicted in class i. Elements are in the range [0, 1].

DataSetMetrics — Metrics aggregated over data set

table

This property is read-only.

Metrics aggregated over the data set, specified as a 1-by-8 table with columns corresponding to these metrics:

• GlobalAccuracy — Ratio of correctly classified images to total images, regardless of class

• MeanAccuracy — Ratio of correctly classified images of each class to total images, averaged over all classes

• Precision — Positive predictive value

• Recall — True positive (TP) rate

• Specificity — True negative (TN) rate

• F1Score — Harmonic mean of precision and recall

• FalsePositiveRate — False positive (FP) rate, or fallout

• FalseNegativeRate — False negative (FN) rate, or miss

ClassMetrics — Class metrics

table

This property is read-only.

Anomaly detection metrics for each class, specified as a 2-by-2 table. The table has rows "Normal" and "Anomaly" and columns corresponding to these metrics:

• Accuracy — Fraction of correctly classified images for each class (normal or anomaly).

• AccuracyPerSubClass — Fraction of correctly classified images for each subclass within the normal and anomaly classes. The value of AccuracyPerSubClass is a table with C rows and one column, where C is the number of subclasses.

Examples

Calculate Metrics for Trained FCDD Anomaly Detector

Load test images and corresponding labels, then create a datastore that reads the test data. The data set consists of grayscale images of handwritten digits 0–9.

[Xtest,gtLabels] = digitTest4DArrayData;
dsTest = arrayDatastore(Xtest,IterationDimension=4);

Load a pretrained FCDD anomaly detector. This detector has been trained to classify the digit 8 as normal and all other digits as anomalies. Therefore, specify the set of anomaly labels as the set of digits between 0 and 9, excluding 8.

load("digit8AnomalyDetector.mat");
anomalyLabels = setdiff(string(0:9),"8");

Classify each test image as normal or anomalous.

predLabels = classify(detector,dsTest);

Calculate metrics for the anomaly detection results.

metrics = evaluateAnomalyDetection(predLabels,gtLabels,anomalyLabels)

Evaluating anomaly detection results
------------------------------------
* Finalizing... Done.
* Data set metrics:

    GlobalAccuracy    MeanAccuracy    Precision    Recall    Specificity    F1Score    FalsePositiveRate    FalseNegativeRate
    ______________    ____________    _________    ______    ___________    _______    _________________    _________________

        0.7662           0.839         0.98971     0.748        0.93        0.85204          0.07                 0.252      

metrics = 
  anomalyDetectionMetrics with properties:

              ConfusionMatrix: [2x2 table]
    NormalizedConfusionMatrix: [2x2 table]
               DataSetMetrics: [1x8 table]
                 ClassMetrics: [2x2 table]

Display the confusion matrix.

cm = metrics.ConfusionMatrix

cm=2×2 table
               Normal    Anomaly
               ______    _______

    Normal       465        35  
    Anomaly     1134      3366  

Version History

Introduced in R2022b