isInNetworkDistribution

R2026b

Determine whether data is within the distribution of the network

Since R2023a

Syntax

tf = isInNetworkDistribution(net,X)

tf = isInNetworkDistribution(discriminator,X)

tf = isInNetworkDistribution(net,X1,...,XN)

tf = isInNetworkDistribution(discriminator,X1,...,XN)

tf = isInNetworkDistribution(___,Name=Value)

Description

Add-On Required: This feature requires the AI Verification Library for Deep Learning Toolbox add-on.

tf = isInNetworkDistribution(net,X) returns a logical array that indicates which observations in X are in-distribution (ID) and which observations are out-of-distribution (OOD). If an observation is ID, then the corresponding element of tf is 1 (true). Otherwise, the corresponding element of tf is 0 (false).

The function computes the distribution confidence score for each observation using the baseline method. For more information, see Softmax-Based Methods. The function classifies any observation with a score less than or equal to the threshold as OOD. To use the default threshold value, use this syntax.

To set the threshold, use the thr name-value argument. Alternatively, use the networkDistributionDiscriminator function to create a discriminator object that automatically finds an optimal threshold and use that as the first input argument instead of net. You can also use the discriminator object to specify a different method to use to compute the distribution confidence scores.

tf = isInNetworkDistribution(discriminator,X) determines which observations in X are ID and which observations are OOD using discriminator. To create a discriminator object, use the networkDistributionDiscriminator function. This syntax uses the threshold stored in the Threshold property of discriminator. Use this syntax to specify additional options for the software to use when it computes the distribution confidence scores and to automatically find a suitable threshold. For example, when creating a discriminator, you can specify whether to use a target true positive or false positive rate to pick the threshold. For more information, see networkDistributionDiscriminator.

tf = isInNetworkDistribution(net,X1,...,XN) determines whether the data is in distribution for networks with multiple inputs using the specified in-memory data.

tf = isInNetworkDistribution(discriminator,X1,...,XN) determines whether the data is in distribution for a discriminator constructed with a network with multiple inputs using the specified in-memory data.

tf = isInNetworkDistribution(___,Name=Value) sets the Threshold and VerbosityLevel options using one or more name-value arguments in addition to the input arguments in previous syntaxes.

Examples

Check if Data is In-Distribution

Load a pretrained classification network.

load("digitsClassificationMLPNetwork.mat")

Load data.

X = digitTrain4DArrayData;

Determine if the data is ID.

tf = isInNetworkDistribution(net,X);

Find the proportion of observations that the function classifies as OOD.

oodProportion = (sum(1-tf)/numel(tf))

oodProportion = 
0.0026

Specify Threshold

Load a pretrained classification network.

load("digitsClassificationMLPNetwork.mat")

Load data.

X = digitTrain4DArrayData;
X = dlarray(X,"SSCB");

Determine if the data is ID using a threshold of 0.9.

tf = isInNetworkDistribution(net,X,Threshold=0.9);

Find the proportion of observations that the function classifies as OOD.

oodProportion = (sum(1-tf)/numel(tf))

oodProportion = 
0

Specify Additional Options Using Discriminator

Load a pretrained classification network.

load("digitsClassificationMLPNetwork.mat")

Load ID data.

X = digitTrain4DArrayData;

Create a discriminator using the networkDistributionDiscriminator function. Set the method to "odin" and the true positive goal to 0.975. The software finds the threshold that satisfies the true positive goal.

method = "odin";
discriminator = networkDistributionDiscriminator(net,X,[],method, ...
    TruePositiveGoal=0.975);

Determine if data is ID.

tf = isInNetworkDistribution(discriminator,X);

Find the true positive rate.

truePositives = sum(tf);
falseNegatives = sum(1-tf); 
truePositiveRate = truePositives/(truePositives + falseNegatives)

truePositiveRate = 
0.9750

Input Arguments

net — Neural network

dlnetwork object

Neural network, specified as a dlnetwork object with a single softmax output.

The software uses the baseline method to compute the distribution confidence scores. To use another method, such as ODIN or energy, specify discriminator as the first input argument. For more information about methods for computing distribution confidence scores, see Distribution Confidence Scores.

For networks without a single softmax layer, create a discriminator object using the networkDistributionDiscriminator function with method set to "hbos" or "kde" (since R2026a) and use this object as the first input argument instead.

discriminator — Distribution discriminator

BaselineDistributionDiscriminator object | ODINDistributionDiscriminator object | EnergyDistributionDiscriminator object | HBOSDistributionDiscriminator object | KDEDistributionDiscriminator object (since R2026a)

Distribution discriminator, specified as a BaselineDistributionDiscriminator, ODINDistributionDiscriminator, EnergyDistributionDiscriminator, HBOSDistributionDiscriminator, or KDEDistributionDiscriminator object. To create this object, use the networkDistributionDiscriminator function.

X — Input data

formatted dlarray | minibatchqueue object | unformatted dlarray (since R2026b) | numeric array (since R2026b) | categorical array (since R2026b) | datastore (since R2026b) | cell array of numeric arrays (since R2026b) | cell array of dlarray objects (since R2026b) | table (since R2026b)

Input data, specified as a formatted or unformatted dlarray object, minibatchqueue, numeric array, categorical array, datastore, cell array, or table.

If you specify a minibatchqueue object, the PreprocessingEnvironment property must be "serial" (default).

Use a minibatchqueue or datastore object for a network with multiple inputs where the data does not fit in memory. If you have data that fits in memory that does not require additional processing, then it is usually easiest to specify the input data as in-memory arrays. For more information, see X1,...,XN.

Before R2026b: If you specify the input as a minibatchqueue object, then it must return an formatted dlarray

X1,...,XN — In-memory data for multi-input network

formatted dlarray | numeric array (since R2026b) | categorical array (since R2026b) | cell array of numeric arrays (since R2026b) | cell array of dlarray objects (since R2026b) | table (since R2026b)

In-memory data for multi-input network, specified as dlarray objects, numeric arrays, categorical arrays, cell arrays, or tables. The input Xi corresponds to the network input which is net.InputNames(i) if net is the first input or discriminator.Network.InputNames(i) if discriminator is the first input.

For multi-input networks, if you have data that fits in memory that does not require additional processing, then it is usually easiest to specify the input data as in-memory arrays. If you want to make predictions with data stored on disk, then specify X as a minibatchqueue or datastore object.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: TruePositiveGoal=0.99,Temperature=10

Threshold — Distribution threshold

0.95 (default) | scalar in the range [0, 1]

Distribution threshold, specified as a scalar in the range [0, 1]. The software uses this value to separate the ID and OOD data.

Dependency

You can only specify this input when the first argument is net. If the first argument is discriminator, then the software uses the threshold stored in the Threshold property of discriminator. For more information, see networkDistributionDiscriminator.

VerbosityLevel — Verbosity level

"off" (default) | "summary" | "detailed"

Verbosity level of the Command Window output, specified as one of these values:

• "off" — Do not display progress information.

• "summary" — Display a summary of the progress information.

• "detailed" — Display detailed information about the progress. This option prints the mini-batch progress.

MiniBatchSize — Size of mini-batches

128 (default) | positive integer

Since R2026b

Size of mini-batches to use for prediction, specified as a positive integer. Larger mini-batch sizes require more memory, but can lead to faster predictions.

To specify padding options, use the SequenceLength name-value argument.

Note

If you specify the input data as a minibatchqueue object, then the isInNetworkDistribution function uses the mini-batch size specified by this argument and not the MiniBatchSize property of the minibatchqueue object. (since R2026b)

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

ExecutionEnvironment — Hardware resource

"auto" (default) | "gpu" | "cpu"

Since R2026b

Hardware resource, specified as one of these values:

• "auto" — Use a GPU if one is available. Otherwise, use the CPU.

• "gpu" — Use the GPU. Using a GPU requires a Parallel Computing Toolbox™ license and a supported GPU device. For information about supported devices, see GPU Computing Requirements (Parallel Computing Toolbox). If Parallel Computing Toolbox or a suitable GPU is not available, then the software returns an error.

• "cpu" — Use the CPU.

Before R2026b: The function uses a CPU.

SequenceLength — Option to pad or truncate sequences

"longest" (default) | "shortest"

Since R2026b

Option to pad or truncate input sequences, specified as one of these values:

• "longest" — Pad sequences in each mini-batch to have the same length as the longest sequence. This option does not discard any data, though padding can introduce noise to the neural network.

• "shortest" — Truncate sequences in each mini-batch to have the same length as the shortest sequence. This option ensures that no padding is added, at the cost of discarding data.

To learn more about the effect of padding and truncating sequences, see Sequence Padding and Truncation.

SequencePaddingDirection — Direction of padding or truncation

"right" (default) | "left"

Since R2026b

Direction of padding or truncation, specified as one of these options:

• "right" — Pad or truncate sequences on the right. The sequences start at the same time step and the software truncates or adds padding to the end of each sequence.

• "left" — Pad or truncate sequences on the left. The software truncates or adds padding to the start of each sequence so that the sequences end at the same time step.

For sequence-to-sequence neural networks (when the recurrent layers output the full sequence), any padding in the first time steps can negatively influence the predictions for the earlier time steps. Right padding helps prevent this issue by ensuring that padding doesn't appear in the initial time steps.

To learn more about the effects of padding and truncating sequences, see Sequence Padding and Truncation.

SequencePaddingValue — Value for padding input sequences

0 (default) | scalar

Since R2026b

Value for padding the input sequences, specified as a scalar.

Do not pad sequences with NaN, because this will cause the function to error.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

CategoricalInputEncoding — Encoding of categorical inputs

"integer" (default) | "one-hot"

Since R2026b

Encoding of categorical inputs, specified as one of these values:

• "integer" — Convert categorical inputs to their integer value. In this case, the network must have one input channel for each of the categorical inputs.

• "one-hot" — Convert categorical inputs to one-hot encoded vectors. In this case, the network must have numCategories channels for each of the categorical inputs, where numCategories is the number of categories of the corresponding categorical input.

InputDataFormats — Description of input data dimensions

"auto" (default) | string array | character vector | cell array of character vectors

Since R2026b

Description of the input data dimensions, specified as a string array, character vector, or cell array of character vectors.

If InputDataFormats is "auto", then the software uses the formats expected by the network input. Otherwise, the software uses the specified formats for the corresponding network input.

A deep learning data format is a string of characters, where each character describes the type of the corresponding data dimension. The characters are:

• "S" — Spatial

• "C" — Channel

• "B" — Batch

• "T" — Time

• "U" — Unspecified

For example, suppose you have an array that represents a batch of sequences where the first, second, and third dimensions correspond to channels, observations, and time steps, respectively. You can describe the data as having the format "CBT" (channel, batch, time).

You can specify multiple dimensions labeled "S" or "U". You can use the labels "C", "B", and "T" at most once each. The software ignores singleton trailing "U" dimensions after the second dimension.

For a neural network with multiple inputs net, specify an array of input data formats, where InputDataFormats(i) corresponds to the input net.InputNames(i).

For more information, see Deep Learning Data Formats.

Data Types: char | string | cell

Output Arguments

tf — True or false result

logical array

True or false result, returned as a numObservations-by-1 logical array with elements set to 1 (true) if the observation is in-distribution and 0 (false) otherwise.

More About

In-Distribution and Out-of-Distribution Data

In-distribution (ID) data refers to any data that you use to construct and train your model. Additionally, any data that is sufficiently similar to the training data is also said to be ID.

Out-of-distribution (OOD) data refers to data that is sufficiently different to the training data. For example, data collected in a different way, at a different time, under different conditions, or for a different task than the data on which the model was originally trained. Models can receive OOD data when you deploy them in an environment other than the one in which you train them. For example, suppose you train a model on clear X-ray images but then deploy the model on images taken with a lower-quality camera.

OOD data detection is important for assigning confidence to the predictions of a network. For more information, see OOD Data Detection.

OOD Data Detection

OOD data detection is a technique for assessing whether the inputs to a network are OOD. For methods that you apply after training, you can construct a discriminator which acts as an additional output of the trained network that classifies an observation as ID or OOD.

The discriminator works by finding a distribution confidence score for an input. You can then specify a threshold. If the score is less than or equal to that threshold, then the input is OOD. Two groups of metrics for computing distribution confidence scores are softmax-based and density-based methods. Softmax-based methods use the softmax layer to compute the scores. Density-based methods use the outputs of layers that you specify to compute the scores. For more information about how to compute distribution confidence scores, see Distribution Confidence Scores.

These images show how a discriminator acts as an additional output of a trained neural network.

Example Data Discriminators

Example of Softmax-Based Discriminator	Example of Density-Based Discriminator
For more information, see Softmax-Based Methods.	For more information, see Density-Based Methods.

Distribution Confidence Scores

Distribution confidence scores are metrics for classifying data as ID or OOD. If an input has a score less than or equal to a threshold value, then you can classify that input as OOD. You can use different techniques for finding the distribution confidence scores.

Softmax-Based Methods

ID data usually corresponds to a higher softmax output than OOD data [1]. Therefore, a method of defining distribution confidence scores is as a function of the softmax scores. These methods are called softmax-based methods. These methods only work for classification networks with a single softmax output.

Let a_i(X) be the input to the softmax layer for class i. The output of the softmax layer for class i is given by this equation:

$$P_i(X;T)=\frac{e^{a_i(X)/T}}{{\displaystyle \sum _{j=1}^C{e}^{a_j(X)/T}}},$$

where C is the number of classes and T is a temperature scaling. When the network predicts the class label of X, the temperature T is set to 1.

The baseline, ODIN, and energy methods each define distribution confidence scores as functions of the softmax input.

• The baseline method [1] uses the maximum of the unscaled output values of the softmax scores:

  $$\text{confidence}(X)=\underset{i}{\max }{P}_i(X;1).$$

• The out-of-distribution detector for neural networks (ODIN) method [2] uses the maximum of the scaled output of the softmax scores:

  $$\text{confidence}(X;T)=\underset{i}{\max }{P}_i(X;T)\text{, }T>0.$$

• The energy method [3] uses the scaled denominator of the softmax output:

  $$\begin{array}{l}\text{energy}(X;T)=-T\log \left({\displaystyle \sum _{j=1}^C{e}^{a_j(X)/T}}\right)\text{, }T>0\\ \text{confidence}(X;T)=-\text{energy}(X;T)\text{, }T>0.\end{array}$$

Density-Based Methods

Density-based methods compute the distribution scores by describing the underlying features learned by the network as probabilistic models. Observations falling into areas of low density correspond to OOD observations.

To model the distributions of the features, you can estimate the probability density function (PDF) for each feature using a histogram or KDE. This technique is based on the histogram-based outlier score (HBOS) method [4]. These methods use a data set of ID data, such as training data, to construct histograms or kernel density estimates representing the density distributions of the ID features. These methods have three stages:

1. Find the principal component features for which to compute the distribution confidence scores:

   1. For each specified layer, find the activations using the n data set observations. Flatten the activations across all dimensions except the batch dimension.

   2. Compute the principal components of the flattened activations matrix. Normalize the eigenvalues such that the largest eigenvalue is 1 and corresponds to the principal component that carries the greatest variance through the layer. Denote the matrix of principal components for layer l by Q^(l).

      The principal components are linear combinations of the activations and represent the features that the software uses to compute the distribution scores. To compute the score, the software uses only the principal components whose eigenvalues are greater than the variance cutoff value σ.

      Note

      The HBOS and KDE algorithms assume that the features are statistically independent. The principal component features are pairwise linearly independent but they can have nonlinear dependencies. To investigate feature dependencies, you can use functions such as corr (Statistics and Machine Learning Toolbox). For an example showing how to investigate feature dependence, see Out-of-Distribution Data Discriminator for YOLO v4 Object Detector. If the features are not statistically independent, then the algorithm can return poor results. Using multiple layers to compute the distribution scores can increase the number of statistically dependent features.

2. For each of the principal component features with an eigenvalue greater than σ, construct a density estimate using either a histogram (HBOS) or KDE.

   • For histograms, the software adjusts the width of the bins to create $$\sqrt{n}$$ bins of approximately equal area, where N is the number of observations. The software then normalizes the bins such that the largest height is 1.

   • For KDE, the software uses the Ramer–Douglas–Peucker algorithm to reduce the number of points. The software linearly interpolates between the remaining KDE points and then normalizes the density to sum to 1.

3. Find the distribution score for an observation by summing the logarithmic probabilities evaluated at the observation for each of the feature density estimates, over each layer.

   Let f^(l)(X) denote the output of layer l for input X. Use the principal components to project the output into a lower dimensional feature space using this equation: $${\widehat{f}}^{(l)}(X)={\left(Q^{(l)}\right)}^T{f}^{(l)}(X)$$.

   Compute the confidence score using this equation:

   $$\begin{array}{l}\text{Score}(X;\sigma )=-{\displaystyle \sum _{l=1}^L\left({\displaystyle \sum _{k=1}^{N^{(l)}(\sigma )}\text{log}\left(h_k^{(l)}\left({\widehat{f}}_k^{(l)}(X)\right)\right)}\right)},\\ \text{confidence}(X;\sigma )=-\text{Score}(X;\sigma ),\end{array}$$

   where N^(l)(σ) is the number of principal components with an eigenvalue greater than σ in layer l, L is the number of layers, and

   • For the HBOS method, h_k^(l) is the normalized histogram height.

   • For the KDE method, h_k^(l) is the reduced KDE value.

   A larger score corresponds to an observation that lies in the areas of higher density. If the density estimate evaluates to 0 for any feature, for example if the observation lies outside of the range of any of the histograms, then the confidence score is -Inf.

   Note

   The distribution scores depend on the properties of the data set the algorithm uses to approximate the PDF.

References

[1] Shalev, Gal, Gabi Shalev, and Joseph Keshet. “A Baseline for Detecting Out-of-Distribution Examples in Image Captioning.” In Proceedings of the 30th ACM International Conference on Multimedia, 4175–84. Lisboa Portugal: ACM, 2022. https://doi.org/10.1145/3503161.3548340.

[2] Shiyu Liang, Yixuan Li, and R. Srikant, “Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks” arXiv:1706.02690 [cs.LG], August 30, 2020, http://arxiv.org/abs/1706.02690.

[3] Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li, “Energy-based Out-of-distribution Detection” arXiv:2010.03759 [cs.LG], April 26, 2021, http://arxiv.org/abs/2010.03759.

[4] Markus Goldstein and Andreas Dengel. "Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm." KI-2012: poster and demo track 9 (2012).

[5] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu, “Generalized Out-of-Distribution Detection: A Survey” August 3, 2022, http://arxiv.org/abs/2110.11334.

[6] Lee, Kimin, Kibok Lee, Honglak Lee, and Jinwoo Shin. “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks.” arXiv, October 27, 2018. http://arxiv.org/abs/1807.03888.

Extended Capabilities

C/C++ Code Generation

Generate C and C++ code using MATLAB® Coder™.

Usage notes and limitations:

• Only the tf = isInNetworkDistribution(discriminator,X) syntax is supported.

• Generated code for isInNetworkDistribution supports single observations only. To process batches of observations, pass each observation individually.

• To load a discriminator object for code generation, use the coder.loadNetworkDistributionDiscriminator function.

• Requires the MATLAB^® Coder™ Interface for Deep Learning support package. If this support package is not installed, use the Add-On Explorer. To open the Add-On Explorer, go to the MATLAB® Toolstrip and click Add-Ons > Get Add-Ons.

GPU Code Generation

Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Usage notes and limitations:

• Only the tf = isInNetworkDistribution(discriminator,X) syntax is supported.

• Generated code for isInNetworkDistribution supports single observations only. To process batches of observations, pass each observation individually.

• To load a discriminator object for code generation, use the coder.loadNetworkDistributionDiscriminator function.

• Requires the GPU Coder™ Interface for Deep Learning support package. If this support package is not installed, use the Add-On Explorer. To open the Add-On Explorer, go to the MATLAB® Toolstrip and click Add-Ons > Get Add-Ons.

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

The isInNetworkDistribution function fully supports GPU acceleration.

By default, the isInNetworkDistribution function uses a GPU is one is available. You can specify the hardware that the isInNetworkDistribution function uses by setting the ExecutionEnvironment argument. (since R2026b)

Before R2026b: This function runs on the GPU if either the network learnable parameters or the input data are gpuArray objects.

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2023a

R2026b: Support for more data types

The isInNetworkDistribution function now supports more data types as input.

Before R2026b, the input data could be a formatted dlarray object, a minibatchqueue object that returns formatted dlarray objects, or [].

Since R2026b, the input data can also be specified as one of these options.

• Unformatted dlarray object

• Numeric array

• Categorical array

• Datastore

• Cell array

• Table

Additionally, the isInNetworkDistribution now supports these name-value arguments:

• MiniBatchSize — Mini-batch size.

• ExecutionEnvironment — Hardware resource.

• SequenceLength — Option to pad or truncate input sequences.

• SequencePaddingDirection — Direction of padding or truncation.

• SequencePaddingValue — Value for padding the input sequences.

• InputDataFormats — Description of the input data dimensions.

• CategoricalInputEncoding — Encoding of categorical inputs.

Note

The isInNetworkDistribution function now has these behavior changes:

• By default, the networkDistributionDiscriminator, isInNetworkDistribution, and distributionScores functions now use a GPU if one is available. Otherwise, they use a CPU. Before R2026b, the functions use a GPU only if the input data or learnable parameters are a gpuArray and otherwise they use a CPU.

• The functions run on batches of data, as specified by the MiniBatchSize name-value option. Before R2026b, the functions use full batches of the data unless you use a minibatchqueue object.

• If you specify the input data as a minibatchqueue object, then the networkDistributionDiscriminator, isInNetworkDistribution, and distributionScores functions use the mini-batch size specified by this argument and not the MiniBatchSize property of the minibatchqueue object. Before R2026b, the batch size is the same as the batch size of the minibatchqueue object.

R2026a: Use kernel density estimation (KDE) discriminator

The isInNetworkDistribution function now supports KDEDistributionDiscriminator objects as input.

The KDE discriminator can produce a smoother approximation of the underlying PDF than HBOS, at the cost of being more computationally expensive to create. For deployment, it is recommended to use the KDE method to reduce changes in OOD classification decisions.

R2023a: Display progress information (June 2023; Version 23.1.1)

Set the VerbosityLevel option to view progress information. The software displays progress information in the Command Window. You can set the VerbosityLevel to "off", "summary", or "detailed".

R2023a: Use minibatchqueue objects (June 2023; Version 23.1.1)

Detect out-of-distribution data using minibatchqueue objects. You can use minibatchequeue objects to create, preprocess, and manage mini-batches of data, and to automatically convert your data to a dlarray object.

R2023a: Code generation support (June 2023; Version 23.1.1)

Generate C or C++ code using MATLAB Coder or generate CUDA^® code for NVIDIA^® GPUs using GPU Coder. For usage notes and limitations, see Extended Capabilities.