In the deep learning custom layer, how to limit the learnable parameters?

Hello, everyone! I have designed a fuzzy system using deep learning toolbox, all the layers are designed using custom layer.

In the layers, some of my learnable parameters should not be zero, i.e. the standard deviation of the gaussian membership function. the fuzzify layer is shown as follows:

classdef fuzzify < nnet.layer.Layer 
%  fuzzify layer, the first layer of fuzzy system
% input : data, centers, sigmas
% output: the fuzzified value of each input data pair
    %#codegen
    properties (Learnable)
        Centers
        Sigmas
    end

    methods
        function layer = fuzzify(Center,Sigma,name)
            layer.Name = name;
            layer.Centers = Center;
            layer.Sigmas = Sigma;
        end

        function [Z,layer] = predict(layer,X)
            % Define layer predict function here.
            X = X';
            [N,M] = size(X);
            C = layer.Centers;
            sigma = layer.Sigmas;          
            nRules = size(C,1);
            Z = zeros(M*nRules,N,'like',X);
            for i = 1:N
                miu = exp(-(X(i,:)-C).^2./(2*(sigma.^2)));
                Z(:,i) = miu(:);
            end
        end

        function [Z,memory] = forward(layer,X)
            % Define layer forward function here.
            X = X';
            [N,M] = size(X);
            C = layer.Centers;
            sigma = layer.Sigmas;
            nRules = size(C,1);
            Z = zeros(M*nRules,N,'like',X);
            for i = 1:N
                miu = exp(-(X(i,:)-C).^2./(2*(sigma.^2)));
                Z(:,i) = miu(:);
            end
            memory=[];
        end
            
        function [dLdX,dLdW1,dLdW2] = backward(layer,X,Z,dLdZ,dLdSin)
            % Define layer backward function here.
            sigma = layer.Sigmas;C = layer.Centers;
            X = X';
            [N,M] = size(X);
            nRules = size(C,1);
            dLdX = zeros(M,N,'like',X);
            dLdW1= zeros(nRules,M,'like',X);
            dLdW2 = zeros(nRules,M,'like',X);
            for i = 1:N
                if isnan(sum(dLdZ(:,i)))
                    continue
                end
                miu = reshape(Z(:,i),nRules,[]);
                dZdX = miu.*(-(X(i,:)-C)./sigma.^2);      
                dZdW1 = miu.*((X(i,:)-C)./sigma.^2);     
                dZdW2 = miu.*((X(i,:)-C).^2./sigma.^3); 
                a = reshape(dLdZ(:,i).*dZdX(:),nRules,[]);
                dLdX(:,i) =  sum(a);
                if ~sum(~isfinite(dLdZ(:,i)))
                    dLdW1 = dLdW1+reshape(dLdZ(:,i).*dZdW1(:),nRules,M);
                    dLdW2 = dLdW2+reshape(dLdZ(:,i).*dZdW2(:),nRules,M);
                end
            end
        end
    end
end

In this code, the learnable parameter 'Sigmas' is the standard deviation of the gaussian membership function. When this parameter being updated, how can I set a threshold for it to prevented it becoming colse to 0.

what I want to do is that: after each iteration, check the value and replace the small number:

threshold = 1e-2;
layer.Sigmas(layer.Sigmas<threshold&layer.Sigmas>0)=threshold;
layer.Sigmas(layer.Sigmas>threshold&layer.Sigmas<0)=-1*threshold;
% this code is only an illustration to show what I want to do

How to realize this? if cannot, Is there any other way？

thanks so much

[Johannes Pitz]

Usually you would want to do something like: sigma = exp(parameter)

Or use the softplus function.

Answer 1

[Christopher Erickson]

@Johannes Pitz' suggestion is excellent. I would add you could use sigmoid to also impose a maximum bound.

However, you can also use dlupdate to use standard gradient descent then impose the bound.

Good luck!