Why peepholeLSTMLayer implemented in a tutorial is much slower than built-in lstmlayer?

Why this implementation of peepholeLSTMLayer https://au.mathworks.com/help/deeplearning/ug/define-custom-recurrent-deep-learning-layer.html is much slower than built-in lstmlayer?

What can be done to speed it up? For example, can it be compiled into a binary code?

Answer 1

[Hiro Yoshino]

I suppose that is because the implementation of interest is a custom model while the built-in LSTM is optimized for computation.

MATLAB has kept improving its performance over the years (see this). So I guess this is also the case with the buil-in capabilities in MATLAB.

As for speeding up, you may choose a CPU for computation of LSTM (See Tips).

You can also see this to speed up your custom trainings.

Hope these help you.

[Artem Lensky]

Hi Hiro,

Thanks for the prompt reply. Yep, I train GRU/LSTM and PeepholeLSTM on a CPU. Peephole is not just slower, it's slower by a factor of 100 compared to standard LSTM. Luckily, this time I don't use custom training loops, it is trained by builtin train function. The model is extremly simple e.g. 1 layer with 8 PeepholeLSTM units. The dimensions of the input signals is 5 by (25k~30k).

1 'sequenceInputLayer' Sequence Input Sequence input with 5 dimensions

2 'rnn_1' peepholeLSTMLayer Peephole LSTM with 8 hidden units

3 'fc' Fully Connected 3 fully connected layer

4 'softmax' Softmax softmax

5 'classoutput' Classification Output crossentropyex

I ran the profiler (just 13 iteration of training) and see below what I've got. Any ideas how I can speed it up? Perhaps updating the tutorial code or compiling it to binary code e.g. mex. There must be something, it is just too slow. Thanks again!