validate

R2026b

Quantize and validate a deep neural network

Syntax

valResults = validate(quantObj,valData)

valResults = validate(quantObj,valData,quantOpts)

Description

Add-On Required: This feature requires the Deep Learning Toolbox Model Compression Library add-on.

valResults = validate(quantObj,valData) quantizes the weights, biases, and activations in the layers of the network, and validates the network specified by dlquantizer object, quantObj, using the data specified by valData.

example

valResults = validate(quantObj,valData,quantOpts) quantizes and validates the network with additional options specified by quantOpts.

example

Examples

collapse all

Quantize a Neural Network for GPU Target

This example uses:

Open Live Script

This example shows how to quantize learnable parameters in the convolution layers of a neural network for GPU and explore the behavior of the quantized network. In this example, you quantize the squeezenet neural network after retraining the network to classify new images. In this example, the memory required for the network is reduced approximately 75% through quantization while the accuracy of the network is not affected.

Load the pretrained network. net is the output network of the Train Deep Learning Network to Classify New Images example.

load squeezedlnetmerch
net

net = 
  dlnetwork with properties:

         Layers: [67×1 nnet.cnn.layer.Layer]
    Connections: [74×2 table]
     Learnables: [52×3 table]
          State: [0×3 table]
     InputNames: {'data'}
    OutputNames: {'prob'}
    Initialized: 1

  View summary with summary.

Define calibration and validation data to use for quantization.

The calibration data is used to collect the dynamic ranges of the weights and biases in the convolution and fully connected layers of the network and the dynamic ranges of the activations in all layers of the network. For the best quantization results, the calibration data must be representative of inputs to the network.

The validation data is used to test the network after quantization to understand the effects of the limited range and precision of the quantized convolution layers in the network.

In this example, use the images in the MerchData data set. Define an augmentedImageDatastore object to resize the data for the network. Then, split the data into calibration and validation data sets.

unzip('MerchData.zip');
imds = imageDatastore('MerchData', ...
    'IncludeSubfolders',true, ...
    'LabelSource','foldernames');
classes = categories(imds.Labels);
[calData, valData] = splitEachLabel(imds, 0.7, 'randomized');
aug_calData = augmentedImageDatastore([227 227], calData);
aug_valData = augmentedImageDatastore([227 227], valData);

Create a dlquantizer object and specify the network to quantize.

dlquantObj = dlquantizer(net);

Specify the GPU target.

quantOpts = dlquantizationOptions(Target='gpu');
quantOpts.MetricFcn = {@(x)hAccuracy(x,net,aug_valData,classes)}

quantOpts = 
  dlquantizationOptions with properties:

   Validation Metric Info
    MetricFcn: {[@(x)hAccuracy(x,net,aug_valData,classes)]}

   Validation Environment Info
       Target: 'gpu'
    Bitstream: ''

Use the calibrate function to exercise the network with sample inputs and collect range information. The calibrate function exercises the network and collects the dynamic ranges of the weights and biases in the convolution and fully connected layers of the network and the dynamic ranges of the activations in all layers of the network. The function returns a table. Each row of the table contains range information for a learnable parameter of the optimized network.

calResults = calibrate(dlquantObj, aug_calData)

calResults = 120×5 table
               'conv1_Weights'               'conv1'    "Weights"    -0.9198    0.8849
                  'conv1_Bias'               'conv1'       "Bias"    -0.0793    0.2634
    'fire2-squeeze1x1_Weights'    'fire2-squeeze1x1'    "Weights"    -1.3800    1.2477
       'fire2-squeeze1x1_Bias'    'fire2-squeeze1x1'       "Bias"    -0.1164    0.2427
     'fire2-expand1x1_Weights'     'fire2-expand1x1'    "Weights"    -0.7406    0.9098
        'fire2-expand1x1_Bias'     'fire2-expand1x1'       "Bias"    -0.0601    0.1460
     'fire2-expand3x3_Weights'     'fire2-expand3x3'    "Weights"    -0.7440    0.6691
        'fire2-expand3x3_Bias'     'fire2-expand3x3'       "Bias"    -0.0518    0.0742
    'fire3-squeeze1x1_Weights'    'fire3-squeeze1x1'    "Weights"    -0.7712    0.6892
       'fire3-squeeze1x1_Bias'    'fire3-squeeze1x1'       "Bias"    -0.1014    0.3267
     'fire3-expand1x1_Weights'     'fire3-expand1x1'    "Weights"    -0.7204    0.9743
        'fire3-expand1x1_Bias'     'fire3-expand1x1'       "Bias"    -0.0670    0.3043
     'fire3-expand3x3_Weights'     'fire3-expand3x3'    "Weights"    -0.6144    0.7741
        'fire3-expand3x3_Bias'     'fire3-expand3x3'       "Bias"    -0.0536    0.1033
      ⋮

Use the validate function to quantize the learnable parameters in the convolution layers of the network and exercise the network. The function uses the metric function defined in the dlquantizationOptions object to compare the results of the network before and after quantization.

valResults = validate(dlquantObj, aug_valData, quantOpts)

valResults = struct with fields:
       NumSamples: 20
    MetricResults: [1×1 struct]
       Statistics: [2×2 table]

Examine the validation output to see the performance of the quantized network.

valResults.MetricResults.Result

ans = 2×2 table
    'Floating-Point'    1
         'Quantized'    1

valResults.Statistics

ans = 2×2 table
    'Floating-Point'    2900268
         'Quantized'     733932

In this example, the memory required for the network was reduced approximately 75% through quantization. The accuracy of the network is not affected.

The weights, biases, and activations of the convolution layers of the network specified in the dlquantizer object now use scaled 8-bit integer data types.

Quantize Network for FPGA Deployment

This example uses:

Open Live Script

Reduce the memory footprint of a deep neural network by quantizing the weights, biases, and activations of convolution layers to 8-bit scaled integer data types. This example shows how to use Deep Learning Toolbox Model Compression Library and Deep Learning HDL Toolbox to deploy the int8 network to a target FPGA board.

Load Pretrained Network

Load the pretrained LogoNet network and analyze the network architecture.

snet = getLogoNetwork;
snet = dag2dlnetwork(snet);
deepNetworkDesigner(snet);

Set random number generator for reproducibility.

rng(0);

Load Data

This example uses the logos_dataset data set. The data set consists of 320 images. Each image is 227-by-227 in size and has three color channels (RGB). Create an augmentedImageDatastore object for calibration and validation.

curDir = pwd;
unzip("logos_dataset.zip");
imageData = imageDatastore(fullfile(curDir,'logos_dataset'),...
'IncludeSubfolders',true,'FileExtensions','.JPG','LabelSource','foldernames');
[calibrationData, validationData] = splitEachLabel(imageData, 0.5,'randomized');

inputSize = [227 227 3];
augCalData = augmentedImageDatastore(inputSize, calibrationData);
augValData = augmentedImageDatastore(inputSize, validationData);

Generate Calibration Result File for the Network

Create a dlquantizer (Deep Learning HDL Toolbox) object and specify the network to quantize. Specify the execution environment as FPGA.

dlQuantObj = dlquantizer(snet,'ExecutionEnvironment',"FPGA");

Use the calibrate (Deep Learning HDL Toolbox) function to exercise the network with sample inputs and collect the range information. The calibrate function collects the dynamic ranges of the weights and biases. The calibrate function returns a table. Each row of the table contains range information for a learnable parameter of the quantized network.

calibrate(dlQuantObj,augCalData)

ans = 35×5 table
    'conv_1_Weights'    'conv_1'    "Weights"    -0.0490    0.0394
       'conv_1_Bias'    'conv_1'       "Bias"          1    1.0028
    'conv_2_Weights'    'conv_2'    "Weights"    -0.0555    0.0619
       'conv_2_Bias'    'conv_2'       "Bias"    -0.0006    0.0023
    'conv_3_Weights'    'conv_3'    "Weights"    -0.0459    0.0469
       'conv_3_Bias'    'conv_3'       "Bias"    -0.0014    0.0015
    'conv_4_Weights'    'conv_4'    "Weights"    -0.0460    0.0510
       'conv_4_Bias'    'conv_4'       "Bias"    -0.0016    0.0038
      'fc_1_Weights'      'fc_1'    "Weights"    -0.0514    0.0543
         'fc_1_Bias'      'fc_1'       "Bias"    -0.0005    0.0008
      'fc_2_Weights'      'fc_2'    "Weights"    -0.0502    0.0516
         'fc_2_Bias'      'fc_2'       "Bias"    -0.0018    0.0019
      'fc_3_Weights'      'fc_3'    "Weights"    -0.0507    0.0468
         'fc_3_Bias'      'fc_3'       "Bias"    -0.0295    0.0249
      ⋮

Create Target Object

Create a target object with a custom name for your target device and an interface to connect your target device to the host computer. Interface options are JTAG and Ethernet. Interface options are JTAG and Ethernet. To use JTAG, install Xilinx Vivado® Design Suite 2022.1. To set the Xilinx Vivado toolpath, enter:

hdlsetuptoolpath('ToolName', 'Xilinx Vivado', 'ToolPath', 'C:\Xilinx\Vivado\2022.1\bin\vivado.bat');

To create the target object, enter:

hTarget = dlhdl.Target('Xilinx','Interface','Ethernet','IPAddress','10.10.10.15');

Alternatively, you can also use the JTAG interface.

% hTarget = dlhdl.Target('Xilinx', 'Interface', 'JTAG');

Create dlQuantizationOptions Object

Create a dlquantizationOptions object. Specify the target bitstream and target board interface. The default metric function is a Top-1 accuracy metric function.

classNames = categories(validationData.Labels);
%options_FPGA = dlquantizationOptions('Bitstream','zcu102_int8','Target',hTarget);
% options_emulation = dlquantizationOptions('Target','host');

To use a custom metric function, specify the metric function in the dlquantizationOptions object.

options_FPGA = dlquantizationOptions(...
    'MetricFcn', {@(x) hComputeModelAccuracyFPGA(x, classNames, validationData)}, ...
    'Bitstream', 'zcu102_int8', ...
    'Target', hTarget);

options_emulation = dlquantizationOptions(...
    'MetricFcn', {@(x) hComputeModelAccuracyFPGA(x, classNames, validationData)});

Validate Quantized Network

Use the validate function to quantize the learnable parameters in the convolution layers of the network. The validate function simulates the quantized network in MATLAB. The validate function uses the metric function defined in the dlquantizationOptions object to compare the results of the single-data-type network object to the results of the quantized network object.

prediction_emulation = dlQuantObj.validate(augValData,options_emulation)

prediction_emulation = struct with fields:
       NumSamples: 160
    MetricResults: [1×1 struct]
       Statistics: []

For validation on an FPGA, the validate function:

Programs the FPGA board by using the output of the compile method and the programming file
Downloads the network weights and biases
Compares the performance of the network before and after quantization

prediction_FPGA = dlQuantObj.validate(augValData,options_FPGA)

### Compiling network for Deep Learning FPGA prototyping ...
### Targeting FPGA bitstream zcu102_int8.
### The network includes the following layers:
1 'imageinput' Image Input 227×227×3 images with 'zerocenter' normalization and 'randfliplr' augmentations (SW Layer)
2 'conv_1' 2-D Convolution 96 5×5×3 convolutions with stride [1 1] and padding [0 0 0 0] (HW Layer)
3 'relu_1' ReLU ReLU (HW Layer)
4 'maxpool_1' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
5 'conv_2' 2-D Convolution 128 3×3×96 convolutions with stride [1 1] and padding [0 0 0 0] (HW Layer)
6 'relu_2' ReLU ReLU (HW Layer)
7 'maxpool_2' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
8 'conv_3' 2-D Convolution 384 3×3×128 convolutions with stride [1 1] and padding [0 0 0 0] (HW Layer)
9 'relu_3' ReLU ReLU (HW Layer)
10 'maxpool_3' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
11 'conv_4' 2-D Convolution 128 3×3×384 convolutions with stride [2 2] and padding [0 0 0 0] (HW Layer)
12 'relu_4' ReLU ReLU (HW Layer)
13 'maxpool_4' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
14 'fc_1' Fully Connected 2048 fully connected layer (HW Layer)
15 'relu_5' ReLU ReLU (HW Layer)
16 'fc_2' Fully Connected 2048 fully connected layer (HW Layer)
17 'relu_6' ReLU ReLU (HW Layer)
18 'fc_3' Fully Connected 32 fully connected layer (HW Layer)
19 'softmax' Softmax softmax (SW Layer)
20 'classoutput' Classification Output crossentropyex with 'adidas' and 31 other classes (SW Layer)

### Notice: The layer 'imageinput' with type 'nnet.cnn.layer.ImageInputLayer' is implemented in software.
### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'classoutput' with type 'nnet.cnn.layer.ClassificationOutputLayer' is implemented in software.
### Compiling layer group: conv_1>>relu_4 ...
### Compiling layer group: conv_1>>relu_4 ... complete.
### Compiling layer group: maxpool_4 ...
### Compiling layer group: maxpool_4 ... complete.
### Compiling layer group: fc_1>>fc_3 ...
### Compiling layer group: fc_1>>fc_3 ... complete.

### Allocating external memory buffers:

offset_name offset_address allocated_space
_______________________ ______________ ________________

"InputDataOffset" "0x00000000" "11.9 MB"
"OutputResultOffset" "0x00be0000" "128.0 kB"
"SchedulerDataOffset" "0x00c00000" "128.0 kB"
"SystemBufferOffset" "0x00c20000" "9.9 MB"
"InstructionDataOffset" "0x01600000" "4.6 MB"
"ConvWeightDataOffset" "0x01aa0000" "8.2 MB"
"FCWeightDataOffset" "0x022e0000" "10.4 MB"
"EndOffset" "0x02d40000" "Total: 45.2 MB"

### Network compilation complete.

### FPGA bitstream programming has been skipped as the same bitstream is already loaded on the target FPGA.
### Deep learning network programming has been skipped as the same network is already loaded on the target FPGA.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Notice: The layer 'imageinput' of type 'ImageInputLayer' is split into an image input layer 'imageinput' and an addition layer 'imageinput_norm' for normalization on hardware.
### The network includes the following layers:
1 'imageinput' Image Input 227×227×3 images with 'zerocenter' normalization and 'randfliplr' augmentations (SW Layer)
2 'conv_1' 2-D Convolution 96 5×5×3 convolutions with stride [1 1] and padding [0 0 0 0] (HW Layer)
3 'relu_1' ReLU ReLU (HW Layer)
4 'maxpool_1' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
5 'conv_2' 2-D Convolution 128 3×3×96 convolutions with stride [1 1] and padding [0 0 0 0] (HW Layer)
6 'relu_2' ReLU ReLU (HW Layer)
7 'maxpool_2' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
8 'conv_3' 2-D Convolution 384 3×3×128 convolutions with stride [1 1] and padding [0 0 0 0] (HW Layer)
9 'relu_3' ReLU ReLU (HW Layer)
10 'maxpool_3' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
11 'conv_4' 2-D Convolution 128 3×3×384 convolutions with stride [2 2] and padding [0 0 0 0] (HW Layer)
12 'relu_4' ReLU ReLU (HW Layer)
13 'maxpool_4' 2-D Max Pooling 3×3 max pooling with stride [2 2] and padding [0 0 0 0] (HW Layer)
14 'fc_1' Fully Connected 2048 fully connected layer (HW Layer)
15 'relu_5' ReLU ReLU (HW Layer)
16 'fc_2' Fully Connected 2048 fully connected layer (HW Layer)
17 'relu_6' ReLU ReLU (HW Layer)
18 'fc_3' Fully Connected 32 fully connected layer (HW Layer)
19 'softmax' Softmax softmax (SW Layer)
20 'classoutput' Classification Output crossentropyex with 'adidas' and 31 other classes (SW Layer)

### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'classoutput' with type 'nnet.cnn.layer.ClassificationOutputLayer' is implemented in software.

Deep Learning Processor Estimator Performance Results

LastFrameLatency(cycles) LastFrameLatency(seconds) FramesNum Total Latency Frames/s
------------- ------------- --------- --------- ---------
Network 39136574 0.17789 1 39136574 5.6
imageinput_norm 216472 0.00098
conv_1 6832680 0.03106
maxpool_1 3705912 0.01685
conv_2 10454501 0.04752
maxpool_2 1173810 0.00534
conv_3 9364533 0.04257
maxpool_3 1229970 0.00559
conv_4 1759348 0.00800
maxpool_4 24450 0.00011
fc_1 2651288 0.01205
fc_2 1696632 0.00771
fc_3 26978 0.00012
* The clock frequency of the DL processor is: 220MHz

### Finished writing input activations.
### Running single input activation.

prediction_FPGA = struct with fields:
       NumSamples: 160
    MetricResults: [1×1 struct]
       Statistics: [2×7 table]

View Performance of Quantized Neural Network

Display the accuracy of the quantized network.

prediction_emulation.MetricResults.Result

ans = 2×2 table
    'Floating-Point'    0.9875
         'Quantized'    0.9875

prediction_FPGA.MetricResults.Result

ans = 2×2 table
    'Floating-Point'    0.9875
         'Quantized'    0.9875

Display the performance of the quantized network in frames per second.

prediction_FPGA.Statistics

ans = 2×7 table
    'Floating-Point'     5.6213    16     4    93.1976    63.9254    15.5952
         'Quantized'    19.4335    64    16    62.3099    50.1096    32.1032

Quantize a Neural Network for CPU Target

This example uses:

Open Live Script

This example shows how to quantize and validate a neural network for a CPU target. This workflow is similar to other execution environments, but before validating you must establish a raspi connection and specify it as target using dlquantizationOptions.

First, load your network. This example uses the pretrained network squeezenet.

load squeezedlnetmerch
net

net = 
  dlnetwork with properties:

         Layers: [67×1 nnet.cnn.layer.Layer]
    Connections: [74×2 table]
     Learnables: [52×3 table]
          State: [0×3 table]
     InputNames: {'data'}
    OutputNames: {'prob'}
    Initialized: 1

  View summary with summary.

Then define your calibration and validation data, calDS and valDS respectively.

unzip('MerchData.zip');
imds = imageDatastore('MerchData', ...
    'IncludeSubfolders',true, ...
    'LabelSource','foldernames');
classes = categories(imds.Labels);
[calData, valData] = splitEachLabel(imds, 0.7, 'randomized');
aug_calData = augmentedImageDatastore([227 227],calData);
aug_valData = augmentedImageDatastore([227 227],valData);

Create the dlquantizer object and specify a CPU execution environment.

dq = dlquantizer(net,'ExecutionEnvironment','CPU')

dq = 
  dlquantizer with properties:

           NetworkObject: [1×1 dlnetwork]
    ExecutionEnvironment: 'CPU'

Calibrate the network.

calResults = calibrate(dq,aug_calData,'UseGPU','off')

calResults = 120×5 table
               "conv1_Weights"               'conv1'    "Weights"    -0.9198    0.8849
                  "conv1_Bias"               'conv1'       "Bias"    -0.0793    0.2634
    "fire2-squeeze1x1_Weights"    'fire2-squeeze1x1'    "Weights"    -1.3800    1.2477
       "fire2-squeeze1x1_Bias"    'fire2-squeeze1x1'       "Bias"    -0.1164    0.2427
     "fire2-expand1x1_Weights"     'fire2-expand1x1'    "Weights"    -0.7406    0.9098
        "fire2-expand1x1_Bias"     'fire2-expand1x1'       "Bias"    -0.0601    0.1460
     "fire2-expand3x3_Weights"     'fire2-expand3x3'    "Weights"    -0.7440    0.6691
        "fire2-expand3x3_Bias"     'fire2-expand3x3'       "Bias"    -0.0518    0.0742
    "fire3-squeeze1x1_Weights"    'fire3-squeeze1x1'    "Weights"    -0.7712    0.6892
       "fire3-squeeze1x1_Bias"    'fire3-squeeze1x1'       "Bias"    -0.1014    0.3267
     "fire3-expand1x1_Weights"     'fire3-expand1x1'    "Weights"    -0.7204    0.9743
        "fire3-expand1x1_Bias"     'fire3-expand1x1'       "Bias"    -0.0670    0.3043
     "fire3-expand3x3_Weights"     'fire3-expand3x3'    "Weights"    -0.6144    0.7741
        "fire3-expand3x3_Bias"     'fire3-expand3x3'       "Bias"    -0.0536    0.1033
      ⋮

Use the Raspberry Pi® Blockset function, raspi, to create a connection to the Raspberry Pi. In the following code, replace:

raspiname with the name or address of your Raspberry Pi
username with your user name
password with your password

% r = raspi('raspiname','username','password')

For example,

r = raspi('gpucoder-raspberrypi-8','pi','matlab')

r = 
  raspi with properties:

         DeviceAddress: 'gpucoder-raspberrypi-8'      
                  Port: 18734                         
             BoardName: 'Raspberry Pi 3 Model B+'     
         AvailableLEDs: {'led0'}                      
  AvailableDigitalPins: [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]
  AvailableSPIChannels: {}                            
     AvailableI2CBuses: {}                            
      AvailableWebcams: {}                            
           I2CBusSpeed:                               
AvailableCANInterfaces: {}                            

  Supported peripherals

Specify raspi object as the target for the quantized network.

opts = dlquantizationOptions('Target',r);
opts.MetricFcn = {@(x)hAccuracy(x,net,aug_valData,classes)}

opts = 
  dlquantizationOptions with properties:

   Validation Metric Info
    MetricFcn: {[@(x)hAccuracy(x,net,aug_valData,classes)]}

   Validation Environment Info
       Target: [1×1 raspi]
    Bitstream: ''

Validate the quantized network with the validate function.

valResults = validate(dq,aug_valData,opts)

### Starting application: 'codegen/lib/validate_predict_int8/pil/validate_predict_int8.elf'
    To terminate execution: clear validate_predict_int8_pil
### Launching application validate_predict_int8.elf...
### Host application produced the following standard output (stdout) and standard error (stderr) messages:

valResults = struct with fields:
       NumSamples: 20
    MetricResults: [1×1 struct]
       Statistics: []

Examine the validation output to see the performance of the quantized network.

valResults.MetricResults.Result

ans = 2×2 table
    'Floating-Point'    1
         'Quantized'    1

Quantize YOLO v3 Object Detector

This example uses:

Open Live Script

This example shows how to quantize a yolov3ObjectDetector (Computer Vision Toolbox) object using preprocessed calibration and validation data.

First, download a pretrained YOLO v3 object detector.

detector = downloadPretrainedNetwork();

This example uses a small labeled data set that contains one or two labeled instances of a vehicle. Many of these images come from the Caltech Cars 1999 and 2001 data sets, created by Pietro Perona and used with permission.

Unzip the vehicle images and load the vehicle ground truth data.

unzip vehicleDatasetImages.zip
data = load('vehicleDatasetGroundTruth.mat');
vehicleDataset = data.vehicleDataset;

Add the full path to the local vehicle data folder.

vehicleDataset.imageFilename = fullfile(pwd, vehicleDataset.imageFilename);

Create an imageDatastore for loading the images and a boxLabelDatastore (Computer Vision Toolbox) for the ground truth bounding boxes.

imds = imageDatastore(vehicleDataset.imageFilename);
blds = boxLabelDatastore(vehicleDataset(:,2));

Use the combine function to combine both the datastores into a CombinedDatastore.

combinedDS = combine(imds, blds);

Split the data into calibration and validation data.

calData = combinedDS.subset(1:32);
valData = combinedDS.subset(33:64);

Use the preprocess (Computer Vision Toolbox) method of yolov3ObjectDetector (Computer Vision Toolbox) object with transform function to prepare the data for calibration and validation.

The transform function returns a TransformedDatastore object.

processedCalData = transform(calData, @(data)preprocess(detector,data));
processedValData = transform(valData, @(data)preprocess(detector,data));

Create the dlquantizer object. When you use the MATLAB execution environment, quantization is performed using the fi fixed-point data type which requires a Fixed-Point Designer™ license.

dq = dlquantizer(detector, 'ExecutionEnvironment', 'MATLAB');

Calibrate the network.

calResults = calibrate(dq, processedCalData,'UseGPU','off')

calResults = 135×5 table
               'conv1_Weights'               'conv1'    "Weights"    -0.9219    0.8569
                  'conv1_Bias'               'conv1'       "Bias"    -0.0963    0.2663
    'fire2-squeeze1x1_Weights'    'fire2-squeeze1x1'    "Weights"    -1.3751    1.2444
       'fire2-squeeze1x1_Bias'    'fire2-squeeze1x1'       "Bias"    -0.1207    0.2310
     'fire2-expand1x1_Weights'     'fire2-expand1x1'    "Weights"    -0.7528    0.9162
        'fire2-expand1x1_Bias'     'fire2-expand1x1'       "Bias"    -0.0593    0.1404
     'fire2-expand3x3_Weights'     'fire2-expand3x3'    "Weights"    -0.7527    0.6774
        'fire2-expand3x3_Bias'     'fire2-expand3x3'       "Bias"    -0.0622    0.0882
    'fire3-squeeze1x1_Weights'    'fire3-squeeze1x1'    "Weights"    -0.7586    0.6877
       'fire3-squeeze1x1_Bias'    'fire3-squeeze1x1'       "Bias"    -0.1021    0.3165
     'fire3-expand1x1_Weights'     'fire3-expand1x1'    "Weights"    -0.7157    0.9768
        'fire3-expand1x1_Bias'     'fire3-expand1x1'       "Bias"    -0.0693    0.3288
     'fire3-expand3x3_Weights'     'fire3-expand3x3'    "Weights"    -0.6008    0.7764
        'fire3-expand3x3_Bias'     'fire3-expand3x3'       "Bias"    -0.0580    0.1123
      ⋮

Validate the quantized network with the validate function.

valResults = validate(dq, processedValData)

valResults = struct with fields:
       NumSamples: 32
    MetricResults: [1×1 struct]
       Statistics: []

function detector = downloadPretrainedNetwork()
   zipFile = 'yolov3SqueezeNetVehicleExample_21aSPKG.zip';
   if ~exist(zipFile,"file")
       pretrainedURL = 'https://ssd.mathworks.com/supportfiles/vision/data/yolov3SqueezeNetVehicleExample_21aSPKG.zip';
       websave(zipFile, pretrainedURL);
   end

   unzip(zipFile);

   pretrained = load("yolov3SqueezeNetVehicleExample_21aSPKG.mat");
   detector = pretrained.detector;
end

Write a Custom Metric Function for Quantized Network Validation

This example uses:

Open Live Script

This example shows how to write a custom metric function to validate a quantized network in the command line.

Load Pretrained Network

Load the pretrained network in dlnetJapaneseVowels.

load dlnetJapaneseVowels

View the network layers. The network is a LSTM network with a single LSTM layer with 100 hidden units.

net.Layers

ans = 
  4×1 Layer array with layers:

     1   'sequenceinput'   Sequence Input    Sequence input with 12 channels
     2   'lstm'            LSTM              LSTM with 100 hidden units
     3   'fc'              Fully Connected   Fully connected layer with output size 9
     4   'softmax'         Softmax           Softmax

View the class names.

classNames

classNames = 9×1 string
    "1"
    "2"
    "3"
    "4"
    "5"
    "6"
    "7"
    "8"
    "9"

Prepare Calibration and Validation Data

Load the Japanese Vowels data set described in [1] and [2] that contains 270 sequences of varying length with 12 features corresponding to LPC cepstrum coefficients and a categorical vector of labels 1, 2, ..., 9. The sequences are matrices with 12 rows (one row for each feature) and a varying number of columns (one column for each time step).

load JapaneseVowelsTrainData

Many deep learning techniques, including the compression techniques available in MATLAB®, require input data of uniform size. You can pad or truncate sequences to be the same length using the padsequences function. In this example, sequences are padded to be of equal length.

calibrationData = padsequences(XTrain,2);
calibrationData = dlarray(calibrationData,"CTB");

Load and pad the Japanese Vowels testing data set with the same method.

load JapaneseVowelsTestData
testData = padsequences(XTest,2);
testData = dlarray(testData,"CTB");

The validate method for quantization only accepts data stores as input for validation data. To ensure the testData data is read as one sample for each batch dimension, write testData to an arrayDatastore with the IterationDimension parameter set to 2.

validationPredictor = arrayDatastore(testData,IterationDimension=2);

Verify one read of the validationPrediction data store returns a dlarray with:

12 channel ('C') dimensions corresponding to the 12 features;
1 batch ('B') dimension corresponding to a single test sequence; and
29 time ('T') dimensions corresponding to the 29 time steps for one padded test sequences.

firstSequence = read(validationPredictor)

firstSequence = 1×1 cell array
    {12×1×29 dlarray}

firstSequenceDimensionLabels = dims(firstSequence{:})

firstSequenceDimensionLabels = 
'CBT'

Combine the padded sequences testData with the labels TTest into one data store for validation data.

validationResponse = arrayDatastore(TTest);
validationData = combine(validationPredictor,validationResponse);

For more information about formatting data for validation, see DATA HANDLING DLQ PAGE.

Quantize Network

Create a dlquantizer object and specify the network to quantize. Set the execution environment to MATLAB. When you use the MATLAB execution environment, quantization is performed using the fi fixed-point data type. Using this data type requires a Fixed-Point Designer™ license.

quantObj = dlquantizer(net,ExecutionEnvironment="MATLAB");

Prepare the network for quantization using prepareNetwork. Network preparation includes converting your network to a dlnetwork object. To later export your quantized network to Simulink, it must be a dlnetwork object quantized for the MATLAB execution environment using MATLAB R2024b or later.

prepareNetwork(quantObj)

Use the calibrate function to exercise the network with the calibration data and collect range statistics for the weights, biases, and activations at each layer.

calResults = calibrate(quantObj,calibrationData);

Use the quantize method to quantize the network object and return a simulatable quantized network.

qNet = quantize(quantObj);

You can use the quantizationDetails function to see that the network is now quantized.

qDetails = quantizationDetails(qNet)

qDetails = struct with fields:
                     IsQuantized: 1
                   TargetLibrary: "none"
             QuantizedLayerNames: [4×1 string]
             QuantizedLearnables: [5×3 table]
    NetworkInputEmbeddedDataType: [1×2 table]
     LayerOutputEmbeddedDataType: [4×2 table]

Write a Custom Metric Function

Define a custom metric function to validate the performance of the quantized network and compare it to the performance of the original floating-point network.

The custom metric function for quantization expects the first input to be the prediction scores for the network. For a classification neural network, the elements of the output correspond to the scores for each class. The order of the scores matches the order of the categories in the training data. For this example, predictionScores is a 9(C) x 370(B) single dlarray. The 9 channel dimension corresponds to the 9 possible classes. The 370 batch dimension corresponds to each of the 370 testing samples.

You can include as many workspace variables as desired as additional inputs to the custom metric function.

For this example, define a custom metric function to compute the accuracy of the network. First convert the prediction scores to predicted labels with scores2label, then calculate the percentage of predicted labels that match the true labels.

function metricOutput = customAccuracy(predictionScores,trueLabels,classNames)

predictedLabels = scores2label(predictionScores,classNames);
metricOutput = mean(trueLabels' == predictedLabels);

end

Validate Quantized Network with Custom Metric Function

To validate the dlnetwork, define a dlquantizationOptions object and specify a custom metric function.

dlquantOpts = dlquantizationOptions; 
dlquantOpts.MetricFcn = {@(x)customAccuracy(x,TTest,classNames)}

dlquantOpts = 
  dlquantizationOptions with properties:

   Validation Metric Info
    MetricFcn: {[@(x)customAccuracy(x,TTest,classNames)]}

   Validation Environment Info
       Target: 'host'
    Bitstream: ''

Use the validate function to compare the results of the network before and after quantization using the validation data set.

valResults = validate(quantObj,validationData,dlquantOpts);

Examine the MetricResults.Result field of the validation output to view the accuracy of the quantized network and the floating-point network.

valResults.MetricResults.Result

ans = 2×2 table
    NetworkImplementation    MetricOutput
    _____________________    ____________

     {'Floating-Point'}        0.85135   
     {'Quantized'     }        0.85946

Validate Quantized Network with Multiple Metric Functions and Non-Scalar Output Types

Define a second custom metric function that computes the percentage of false positives and false negatives and the macro-averaged error rate value given predictions and targets. This equation defines the macro error rate:

$m a c r o a v e r a g e e r r o r r a t e = \frac{1}{K} \sum_{i = 1}^{K} \frac{{F P}_{i} + {F N}_{i}}{{T P}_{i} + {T N}_{i} + {F P}_{i} + {F N}_{i}}$

where $T P_{i}$ , $T N_{i}$ , $F P_{i}$ , and $F N_{i}$ represent the number of true positives, true negatives, false positives, and false negatives, respectively, in class $i$ and $K$ is the number of classes.

function metricOutput = customMacroAverageError(predictionScores,testLabels,classNames)

% One hot encode the true labels of the test data.
encodedTrueLabels = onehotencode(testLabels,2);

% Convert predictionScores to labels and one hot encode the predicted
% labels
predictedLabels = scores2label(predictionScores,classNames);
encodedPredictions = onehotencode(predictedLabels,1)';

% Find the TP, FP, FN, and TN for each class.
TP = sum(encodedPredictions & encodedTrueLabels, 1);
FP = sum(encodedPredictions & ~encodedTrueLabels, 1);
FN = sum(~encodedPredictions & encodedTrueLabels, 1);
TN = sum(~encodedPredictions & ~encodedTrueLabels, 1);

% Return the number of false positives, number of 
% false negatives, and macro-average error rate.
falsePositives = sum(FP);
falseNegatives = sum(FN);
macroAverageError = mean((FP + FN) ./ (TP + TN + FP + FN));
metricOutput = table(falsePositives,falseNegatives,macroAverageError, ...
    VariableNames={'Number False Positives';'Number False Negatives';'Macro Averaged Error'});
end

Define a new dlquantizationOptions object and specify both custom metric functions.

dlquantOptsMultiple = dlquantizationOptions; 
dlquantOptsMultiple.MetricFcn = {@(x)customAccuracy(x,TTest,classNames),@(x)customMacroAverageError(x,TTest,classNames)}

dlquantOptsMultiple = 
  dlquantizationOptions with properties:

   Validation Metric Info
    MetricFcn: {[@(x)customAccuracy(x,TTest,classNames)]  [@(x)customMacroAverageError(x,TTest,classNames)]}

   Validation Environment Info
       Target: 'host'
    Bitstream: ''

Use the validate function to compare the results of the network before and after quantization using the validation data set.

valResultsMultiple = validate(quantObj,validationData,dlquantOptsMultiple);

Examine the MetricResults.Result field of the validation output to view the results of the custom metric function for the quantized network and the floating-point network.

valResultsMultiple.MetricResults.Result

ans = 2×2 table
    NetworkImplementation    MetricOutput
    _____________________    ____________

     {'Floating-Point'}        0.85135   
     {'Quantized'     }        0.85946

ans = 2×2 table
    NetworkImplementation                                  MetricOutput                              
    _____________________    ________________________________________________________________________

                             Number False Positives    Number False Negatives    Macro Averaged Error
                             ______________________    ______________________    ____________________
                                                                                                     
     {'Floating-Point'}                55                        55                    0.033033      
     {'Quantized'     }                52                        52                    0.031231

Bibliography

M. Kudo, J. Toyama, and M. Shimbo. "Multidimensional Curve Classification Using Passing-Through Regions." Pattern Recognition Letters. Vol. 20, No. 11–13, pages 1103–1111.
UCI Machine Learning Repository: Japanese Vowels Dataset. https://archive.ics.uci.edu/ml/datasets/Japanese+Vowels

Validate Quantized Network on FPGA Using Custom Bitstream

This example uses:

Open Live Script

Validate a dlquantizer object on a target FPGA board using a custom bitstream, and compare the results of validation using two custom int8 bitstreams with different thread counts. In this example, you will quantize a pretrained network, generate custom bitstreams, and validate the quantized network using the custom bitstreams.

Quantize Pretrained Network

Load the pretrained digits network.

snet = getDigitsNetwork;

Load image data for quantization and create calibration and validation datastores. For more information on the data used in this example, see Data Sets for Deep Learning.

dataFolder = fullfile(toolboxdir('nnet'),'nndemos','nndatasets','DigitDataset');
imds = imageDatastore(dataFolder, 'IncludeSubfolders',true,'LabelSource','foldernames');

[calData,valData] = splitEachLabel(imds,0.7,'randomized');
calData_subset = calData.subset(1:20);
valData_subset = valData.subset(1:6);

Quantize the network using a dlquantizer object. Specify FPGA as the execution environment.

dq = dlquantizer(snet,'ExecutionEnvironment','FPGA');
dq.calibrate(calData_subset);

To validate the network on a target FPGA board, specify a dlhdl.Target object. This example uses a Xilinx™ ZCU102 ZU9EG device.

hTarget = dlhdl.Target('Xilinx','Interface','JTAG');

Generate Custom Bitstreams

To compare the performance of custom bitstreams, generate two bitstreams with different configurations. The bitstreams used in this example are customized to show the performance and resource utilization difference between int8 bitstreams with different processor thread counts for the convolution and fully connected modules on the Xilinx™ ZCU102 ZU9EG device.

Generating a bitstream can take several hours. Before generating a bitstream, you can use the optimizeConfigurationForNetwork (Deep Learning HDL Toolbox) method to modify the processor configuration to meet the requirements of your network and target device. For a list of existing bitstreams, see Use Deep Learning on FPGA Bitstreams (Deep Learning HDL Toolbox).

Use a dlhdl.ProcessorConfig object to specify the processor parameters for your custom bitstream. For a quantized network, specify the processor data type as 'int8'. For an int8 processor, the default values assigned to ConvThreadNumber and FCThreadNumber are 16 and 4, respectively. Generate the bitstream using the dlhdl.buildProcessor function. For more information about how to generate a custom bitstream, see Generate Custom Bitstream (Deep Learning HDL Toolbox).

hPCNew = dlhdl.ProcessorConfig
hPCNew.ProcessorDataType = 'int8';
dlhdl.buildProcessor(hPCNew);

Save the generated bitstream as 'custom_int8.bit'. After saving the generated bitstream, use the same dlhdl.ProcessorConfig object to generate a second bitstream. Increase the ConvThreadNumber to 64 and FCThreadNumber to 16.

hPCNew.setModuleProperty('conv','ConvThreadNumber',64);
hPCNew.setModuleProperty('fc','FCThreadNumber',16);
dlhdl.buildProcessor(hPCNew);

Save the new generated bitstream as 'custom_int8_incThread.bit'.

Validate Using Generated Bitstreams

Validate the quantized network on the target device using the first generated bitstream, 'custom_int8.bit'. Specify the bitstream to use for validation using a dlquantizationOptions object. If the bitstream is not in your working directory, specify the full path to the file.

dlquantOpts_custom_int8 = dlquantizationOptions('Bitstream','custom_int8.bit','Target',hTarget);
valResults_custom_int8 = dq.validate(valData_subset,dlquantOpts_custom_int8);

### Compiling network for Deep Learning FPGA prototyping ...
### Targeting FPGA bitstream custom_int8.bit.
### Optimizing network: Fused 'nnet.cnn.layer.BatchNormalizationLayer' into 'nnet.cnn.layer.Convolution2DLayer'
### The network includes the following layers:
     1   'imageinput'    Image Input             28×28×1 images with 'zerocenter' normalization                (SW Layer)
     2   'conv_1'        2-D Convolution         8 3×3×1 convolutions with stride [1  1] and padding 'same'    (HW Layer)
     3   'relu_1'        ReLU                    ReLU                                                          (HW Layer)
     4   'maxpool_1'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     5   'conv_2'        2-D Convolution         16 3×3×8 convolutions with stride [1  1] and padding 'same'   (HW Layer)
     6   'relu_2'        ReLU                    ReLU                                                          (HW Layer)
     7   'maxpool_2'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     8   'conv_3'        2-D Convolution         32 3×3×16 convolutions with stride [1  1] and padding 'same'  (HW Layer)
     9   'relu_3'        ReLU                    ReLU                                                          (HW Layer)
    10   'fc'            Fully Connected         10 fully connected layer                                      (HW Layer)
    11   'softmax'       Softmax                 softmax                                                       (SW Layer)
    12   'classoutput'   Classification Output   crossentropyex with '0' and 9 other classes                   (SW Layer)
                                                                                                             
### Notice: The layer 'imageinput' with type 'nnet.cnn.layer.ImageInputLayer' is implemented in software.
### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'classoutput' with type 'nnet.cnn.layer.ClassificationOutputLayer' is implemented in software.
### Compiling layer group: conv_1>>maxpool_2 ...
### Compiling layer group: conv_1>>maxpool_2 ... complete.
### Compiling layer group: conv_3>>relu_3 ...
### Compiling layer group: conv_3>>relu_3 ... complete.
### Compiling layer group: fc ...
### Compiling layer group: fc ... complete.

### Allocating external memory buffers:

          offset_name          offset_address     allocated_space 
    _______________________    ______________    _________________

    "InputDataOffset"           "0x00000000"     "184.0 kB"       
    "OutputResultOffset"        "0x0002e000"     "4.0 kB"         
    "SchedulerDataOffset"       "0x0002f000"     "8.0 kB"         
    "SystemBufferOffset"        "0x00031000"     "36.0 kB"        
    "InstructionDataOffset"     "0x0003a000"     "16.0 kB"        
    "ConvWeightDataOffset"      "0x0003e000"     "8.0 kB"         
    "FCWeightDataOffset"        "0x00040000"     "28.0 kB"        
    "EndOffset"                 "0x00047000"     "Total: 284.0 kB"

### Network compilation complete.

### Programming FPGA Bitstream using JTAG...
### Programming the FPGA bitstream has been completed successfully.
### Loading weights to Conv Processor.
### Conv Weights loaded. Current time is 16-Jan-2024 15:08:59
### Loading weights to FC Processor.
### FC Weights loaded. Current time is 16-Jan-2024 15:08:59
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Optimizing network: Fused 'nnet.cnn.layer.BatchNormalizationLayer' into 'nnet.cnn.layer.Convolution2DLayer'
### Notice: The layer 'imageinput' of type 'ImageInputLayer' is split into an image input layer 'imageinput' and an addition layer 'imageinput_norm' for normalization on hardware.
### The network includes the following layers:
     1   'imageinput'    Image Input             28×28×1 images with 'zerocenter' normalization                (SW Layer)
     2   'conv_1'        2-D Convolution         8 3×3×1 convolutions with stride [1  1] and padding 'same'    (HW Layer)
     3   'relu_1'        ReLU                    ReLU                                                          (HW Layer)
     4   'maxpool_1'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     5   'conv_2'        2-D Convolution         16 3×3×8 convolutions with stride [1  1] and padding 'same'   (HW Layer)
     6   'relu_2'        ReLU                    ReLU                                                          (HW Layer)
     7   'maxpool_2'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     8   'conv_3'        2-D Convolution         32 3×3×16 convolutions with stride [1  1] and padding 'same'  (HW Layer)
     9   'relu_3'        ReLU                    ReLU                                                          (HW Layer)
    10   'fc'            Fully Connected         10 fully connected layer                                      (HW Layer)
    11   'softmax'       Softmax                 softmax                                                       (SW Layer)
    12   'classoutput'   Classification Output   crossentropyex with '0' and 9 other classes                   (SW Layer)
                                                                                                             
### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'classoutput' with type 'nnet.cnn.layer.ClassificationOutputLayer' is implemented in software.


              Deep Learning Processor Estimator Performance Results

                   LastFrameLatency(cycles)   LastFrameLatency(seconds)       FramesNum      Total Latency     Frames/s
                         -------------             -------------              ---------        ---------       ---------
Network                      22272                  0.00011                       1              2e+04           8979.7
    imageinput_norm           4236                  0.00002 
    conv_1                    4494                  0.00002 
    maxpool_1                 2999                  0.00001 
    conv_2                    2455                  0.00001 
    maxpool_2                 2388                  0.00001 
    conv_3                    2354                  0.00001 
    fc                        3346                  0.00002 
 * The clock frequency of the DL processor is: 200MHz


### Finished writing input activations.
### Running single input activation.

Validate the quantized network on the target device using the second generated bitstream, 'custom_int8_incThread.bit'.

dlquantOpts_custom_incThread = dlquantizationOptions('Bitstream','custom_int8_incThread.bit','Target',hTarget);
valResults_custom_incThread = dq.validate(valData_subset,dlquantOpts_custom_incThread);

### Compiling network for Deep Learning FPGA prototyping ...
### Targeting FPGA bitstream custom_int8_incThread.bit.
### Optimizing network: Fused 'nnet.cnn.layer.BatchNormalizationLayer' into 'nnet.cnn.layer.Convolution2DLayer'
### The network includes the following layers:
     1   'imageinput'    Image Input             28×28×1 images with 'zerocenter' normalization                (SW Layer)
     2   'conv_1'        2-D Convolution         8 3×3×1 convolutions with stride [1  1] and padding 'same'    (HW Layer)
     3   'relu_1'        ReLU                    ReLU                                                          (HW Layer)
     4   'maxpool_1'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     5   'conv_2'        2-D Convolution         16 3×3×8 convolutions with stride [1  1] and padding 'same'   (HW Layer)
     6   'relu_2'        ReLU                    ReLU                                                          (HW Layer)
     7   'maxpool_2'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     8   'conv_3'        2-D Convolution         32 3×3×16 convolutions with stride [1  1] and padding 'same'  (HW Layer)
     9   'relu_3'        ReLU                    ReLU                                                          (HW Layer)
    10   'fc'            Fully Connected         10 fully connected layer                                      (HW Layer)
    11   'softmax'       Softmax                 softmax                                                       (SW Layer)
    12   'classoutput'   Classification Output   crossentropyex with '0' and 9 other classes                   (SW Layer)
                                                                                                             
### Notice: The layer 'imageinput' with type 'nnet.cnn.layer.ImageInputLayer' is implemented in software.
### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'classoutput' with type 'nnet.cnn.layer.ClassificationOutputLayer' is implemented in software.
### Compiling layer group: conv_1>>maxpool_2 ...
### Compiling layer group: conv_1>>maxpool_2 ... complete.
### Compiling layer group: conv_3>>relu_3 ...
### Compiling layer group: conv_3>>relu_3 ... complete.
### Compiling layer group: fc ...
### Compiling layer group: fc ... complete.

### Allocating external memory buffers:

          offset_name          offset_address     allocated_space 
    _______________________    ______________    _________________

    "InputDataOffset"           "0x00000000"     "92.0 kB"        
    "OutputResultOffset"        "0x00017000"     "4.0 kB"         
    "SchedulerDataOffset"       "0x00018000"     "36.0 kB"        
    "SystemBufferOffset"        "0x00021000"     "36.0 kB"        
    "InstructionDataOffset"     "0x0002a000"     "28.0 kB"        
    "ConvWeightDataOffset"      "0x00031000"     "8.0 kB"         
    "FCWeightDataOffset"        "0x00033000"     "20.0 kB"        
    "EndOffset"                 "0x00038000"     "Total: 224.0 kB"

### Network compilation complete.

### Programming FPGA Bitstream using JTAG...
### Programming the FPGA bitstream has been completed successfully.
### Loading weights to Conv Processor.
### Conv Weights loaded. Current time is 16-Jan-2024 15:10:57
### Loading weights to FC Processor.
### FC Weights loaded. Current time is 16-Jan-2024 15:10:57
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Finished writing input activations.
### Running single input activation.
### Optimizing network: Fused 'nnet.cnn.layer.BatchNormalizationLayer' into 'nnet.cnn.layer.Convolution2DLayer'
### Notice: The layer 'imageinput' of type 'ImageInputLayer' is split into an image input layer 'imageinput' and an addition layer 'imageinput_norm' for normalization on hardware.
### The network includes the following layers:
     1   'imageinput'    Image Input             28×28×1 images with 'zerocenter' normalization                (SW Layer)
     2   'conv_1'        2-D Convolution         8 3×3×1 convolutions with stride [1  1] and padding 'same'    (HW Layer)
     3   'relu_1'        ReLU                    ReLU                                                          (HW Layer)
     4   'maxpool_1'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     5   'conv_2'        2-D Convolution         16 3×3×8 convolutions with stride [1  1] and padding 'same'   (HW Layer)
     6   'relu_2'        ReLU                    ReLU                                                          (HW Layer)
     7   'maxpool_2'     2-D Max Pooling         2×2 max pooling with stride [2  2] and padding [0  0  0  0]   (HW Layer)
     8   'conv_3'        2-D Convolution         32 3×3×16 convolutions with stride [1  1] and padding 'same'  (HW Layer)
     9   'relu_3'        ReLU                    ReLU                                                          (HW Layer)
    10   'fc'            Fully Connected         10 fully connected layer                                      (HW Layer)
    11   'softmax'       Softmax                 softmax                                                       (SW Layer)
    12   'classoutput'   Classification Output   crossentropyex with '0' and 9 other classes                   (SW Layer)
                                                                                                             
### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'classoutput' with type 'nnet.cnn.layer.ClassificationOutputLayer' is implemented in software.


              Deep Learning Processor Estimator Performance Results

                   LastFrameLatency(cycles)   LastFrameLatency(seconds)       FramesNum      Total Latency     Frames/s
                         -------------             -------------              ---------        ---------       ---------
Network                      41020                  0.00021                       1              41020           4875.7
    imageinput_norm           4236                  0.00002 
    conv_1                    6683                  0.00003 
    maxpool_1                 5804                  0.00003 
    conv_2                    5509                  0.00003 
    maxpool_2                 4582                  0.00002 
    conv_3                    5905                  0.00003 
    fc                        8301                  0.00004 
 * The clock frequency of the DL processor is: 200MHz


### Finished writing input activations.
### Running single input activation.

Compare Validation Output

Compare the validation results from both bitstreams. For these bitstream configurations, increasing the number of threads used for convolution and fully connected layers increases the number of frames per second as well as the resource utilization. For more information on how to optimize your processor configuration based on the resource requirements of your hardware, see Estimate Resource Utilization for Custom Processor Configuration (Deep Learning HDL Toolbox).

valResults_custom_int8.Statistics

ans = 2×7 table
    'Floating-Point'    4875.6704    16    4    78.8523788674839    55.7565789473684    15.4365079365079
         'Quantized'    6418.8972    16    4     36.685274372446    47.7521929824561    10.5952380952381

valResults_custom_incThread.Statistics

ans = 2×7 table
    'Floating-Point'     8979.6835    64    16    264.974970811442    61.4583333333333    52.5793650793651
         'Quantized'    12126.3566    64    16    61.4561441914769     49.671052631579    32.0238095238095

Input Arguments

collapse all

`quantObj` — Network to quantize
`dlquantizer` object

Network to quantize, specified as a dlquantizer object.

`valData` — Data to use for validation of quantized network
`imageDatastore` object | `augmentedImageDatastore` object | `pixelLabelImageDatastore` object | `CombinedDatastore` object | `TransformedDatastore` object

Data to use for validation of quantized network, specified as an imageDatastore object, an augmentedImageDatastore object, a pixelLabelImageDatastore (Computer Vision Toolbox) object, a CombinedDatastore object, or a TransformedDatastore object.

You must preprocess the data used for validation of a quantized yolov3ObjectDetector (Computer Vision Toolbox) object using the preprocess (Computer Vision Toolbox) function. For an example of using preprocessed data for validation of a yolov3ObjectDetector, see Quantize YOLO v3 Object Detector.

validate accepts a CombinedDatastore or TransformedDatastore object as input data for validating quantized yolov3ObjectDetector and yolov4ObjectDetector objects. The CombinedDatastore and TransformedDatastore used for validation must contain an imageDatastore or augmentedImageDatastore as the first datastore and a boxLabelDatastore as the second datastore. For more information on valid datastores, see Prepare Data for Quantizing Networks.

`quantOpts` — Options for quantizing network
`dlquantizationOptions` object

Options for quantizing the network, specified as a dlquantizationOptions object.

Output Arguments

collapse all

`valResults` — Performance of quantized network
struct

Performance of quantized network, returned as a struct. The struct contains these fields.

NumSamples — The number of sample inputs used to validate the network, specified by valData.

MetricResults — Struct containing results of the metric function defined in the dlquantizationOptions object. When more than one metric function is specified in the dlquantizationOptions object, MetricResults is an array of structs.

MetricResults contains these fields:

Field Description

MetricFunction Metric function used to determine the performance of the quantized network, specified in the dlquantizationOptions object.

Field	Description
`MetricFunction`	Metric function used to determine the performance of the quantized network, specified in the `dlquantizationOptions` object.
`Result`	Table indicating the results of the metric function before and after quantization. The first row in the table, `'Floating-Point'`, contains information for the original floating-point implementation. The second row, `'Quantized'`, contains information for the quantized implementation. The output of the metric function is displayed in the `MetricOutput` column.

Result

Table indicating the results of the metric function before and after quantization.

The first row in the table, 'Floating-Point', contains information for the original floating-point implementation. The second row, 'Quantized', contains information for the quantized implementation. The output of the metric function is displayed in the MetricOutput column.

Statistics — Table indicating the learnable parameter memory used, in bytes, by the original floating-point implementation of the network and the quantized implementation.
When the ExecutionEnvironment for the dlquantizer object is set to FPGA, the Statistics table is a table indicating these values for the original floating-point and quantized network implementations:
- Frames per second
- Number of convolution threads
- Number of fully connected threads
- Lookup table (LUT) resource utilization percentage
- Block RAM resource utilization percentage
- DSP resource utilization percentage
The Statistics table will be empty when the Target property of dlquantizationOptions is set to 'host'.

Limitations

Validation on target hardware for CPU, FPGA, and GPU execution environments is not supported in MATLAB^® Online™. For FPGA and GPU execution environments, validation can be performed through emulation on the MATLAB Online host.

Algorithms

The validate function determines the default metric function to use for the validation based on the type of network that is being quantized.

Type of Network	Metric Function
Classification	Top-1 Accuracy — Accuracy of the network
Object Detection	Average Precision — Average precision over all detection results. See `evaluateObjectDetection` (Computer Vision Toolbox).
Regression	MSE — Mean squared error of the network
Semantic Segmentation	`evaluateSemanticSegmentation` (Computer Vision Toolbox) — Evaluate semantic segmentation data set against ground truth
Single Shot Detector (SSD)	WeightedIOU — Average IoU of each class, weighted by the number of pixels in that class

Version History

Introduced in R2020a

expand all

R2026b: Quantized `dlnetwork` objects designed for fixed-point inputs and outputs

Quantized dlnetwork objects produced by the quantize function with the MATLAB execution environment now accept fixed-point input data for inference. If your input data does not already match the expected type, the quantized network casts input data to this data type before passing it to the input layer. In releases before R2026b, quantized networks only accepted floating-point input data.

Outputs of layers are represented in fixed-point when the data is passed between quantized layers. In releases before R2026b, the network output for a quantized dlnetwork object was always single-precision floating-point data.

Use the quantizationDetails function to inspect the fixed-point data types for network inputs and layer outputs.

R2026a: Image input data must match dimension of network input layer for detector network objects

For yolov2ObjectDetector (Computer Vision Toolbox), yolov3ObjectDetector (Computer Vision Toolbox), yolov4ObjectDetector (Computer Vision Toolbox), and ssdObjectDetector (Computer Vision Toolbox) network objects, the validate function now requires image input data to match the data dimensions expected by the input layer of your network. In previous releases, the validate function automatically resized image data to match the input of the network using imresize.

R2024a: Validate with array datastores

The validate function supports arrayDatastore objects as validation data input.

R2024a: Validate `dlnetwork` objects on FPGA

You can validate a dlnetwork on a target FPGA board using the validate function.

R2023b: Validate behavior of networks through emulation

You can validate the behavior of a quantized network through emulation for workflows with the ExecutionEnvironment option set to "FPGA", "GPU", or "MATLAB". Emulation does not require target hardware.

To validate through emulation in the command line, set the Target property of dlquantizationOptions to "host" when you call the validate function.

R2022a: Validate the performance of quantized network for CPU target

You can now use the dlquantizer object and the validate function to quantize a network and generate code for CPU targets.

validate

Syntax

Description

Examples

Quantize a Neural Network for GPU Target

Quantize Network for FPGA Deployment

Quantize a Neural Network for CPU Target

Quantize YOLO v3 Object Detector

Write a Custom Metric Function for Quantized Network Validation

Validate Quantized Network on FPGA Using Custom Bitstream

Input Arguments

`quantObj` — Network to quantize
`dlquantizer` object

`valData` — Data to use for validation of quantized network
`imageDatastore` object | `augmentedImageDatastore` object | `pixelLabelImageDatastore` object | `CombinedDatastore` object | `TransformedDatastore` object

`quantOpts` — Options for quantizing network
`dlquantizationOptions` object

Output Arguments

`valResults` — Performance of quantized network
struct

Limitations

Algorithms

Version History

R2026b: Quantized `dlnetwork` objects designed for fixed-point inputs and outputs

R2026a: Image input data must match dimension of network input layer for detector network objects

R2024a: Validate with array datastores

R2024a: Validate `dlnetwork` objects on FPGA

R2023b: Validate behavior of networks through emulation

R2022a: Validate the performance of quantized network for CPU target

See Also

Apps

Functions

Topics

validate

Syntax

Description

Examples

Quantize a Neural Network for GPU Target

Quantize Network for FPGA Deployment

Quantize a Neural Network for CPU Target

Quantize YOLO v3 Object Detector

Write a Custom Metric Function for Quantized Network Validation

Validate Quantized Network on FPGA Using Custom Bitstream

Input Arguments

quantObj — Network to quantize dlquantizer object

valData — Data to use for validation of quantized network imageDatastore object | augmentedImageDatastore object | pixelLabelImageDatastore object | CombinedDatastore object | TransformedDatastore object

quantOpts — Options for quantizing network dlquantizationOptions object

Output Arguments

valResults — Performance of quantized network struct

Limitations

Algorithms

Version History

R2026b: Quantized dlnetwork objects designed for fixed-point inputs and outputs

R2026a: Image input data must match dimension of network input layer for detector network objects

R2024a: Validate with array datastores

R2024a: Validate dlnetwork objects on FPGA

R2023b: Validate behavior of networks through emulation

R2022a: Validate the performance of quantized network for CPU target

See Also

Apps

Functions

Topics

`quantObj` — Network to quantize
`dlquantizer` object

`valData` — Data to use for validation of quantized network
`imageDatastore` object | `augmentedImageDatastore` object | `pixelLabelImageDatastore` object | `CombinedDatastore` object | `TransformedDatastore` object

`quantOpts` — Options for quantizing network
`dlquantizationOptions` object

`valResults` — Performance of quantized network
struct

R2026b: Quantized `dlnetwork` objects designed for fixed-point inputs and outputs

R2024a: Validate `dlnetwork` objects on FPGA