Is it possible (yet) to implement a maxout activation "layer" in 2017b Deep Learning network?

Maxout is an activation function that includes RELU and “leaky” RELUs as special cases, basically allowing for piecewise linear (planar/hyperplanar) activation functions. They seem to work better than either in a number of cases. Here’s a reference: Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y. (2013). Maxout networks. arXiv preprint arXiv:1302.4389. https://arxiv.org/abs/1302.4389

Ultimately I’m interested in playing with architectures like this one, which use maxouts extensively:

Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Bengio, C. L. Y., & Courville, A. (2017). Towards end-to-end speech recognition with deep convolutional neural networks. arXiv preprint arXiv:1701.02720. Speech recognition using convolutional nets with maxout activation. https://arxiv.org/abs/1701.02720

But I simply can’t see any way to fake a maxout activation in a convolutional network framework in 2017b. While I’m a Matlab vet (since Version 4, I think), I’m a total newbie to Matlab deep learning networks, so maybe I’m missing something. Any suggestions greatly appreciated.

-Terry Nearey

Answer 1

[Pankaj Wasnik]

Hi, You can try using https://github.com/yechengxi/LightNet, which is a bit simpler cnn toolbox where you can debug easily also it's easier to understand. You can try to implement the maxout layer by yourself. I am also trying the same. If I finish before you, I will share the code.

Regards, Pankaj Wasnik

Answer 2

[Gayathri]

Hi @Terrance Nearey,

To implement "Maxout" activation layer, we will have to implement the functionality as a custom layer.

The following code is a minimal working example of a custom “Maxout” layer:

classdef MaxoutLayer < nnet.layer.Layer
    properties
        NumPieces
    end
    
    methods
        function layer = MaxoutLayer(numPieces, name)
            layer.Name = name;
            layer.Description = ['Maxout with ', num2str(numPieces), ' pieces'];
            layer.NumPieces = numPieces;
        end
        
        function Z = predict(layer, X)
            inputSize = size(X);
            H = inputSize(1);
            W = inputSize(2);
            N = inputSize(4);
            C = inputSize(3) / layer.NumPieces;
            X = reshape(X, H, W, C, layer.NumPieces, N);
            Z = max(X, [], 4);
        end
    end
end

Please make sure that the convolutional layer feeding into the “Maxout” layer produces a number of channels divisible by "NumPieces", as required by the reshape logic.

These are some documentation links which can be useful:

• Layer Graph - https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layergraph.html
• Custom Deep Learning layers - https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layers.html

Hope you find this useful!