Recurrent Nets
Sequences and time series need a network that remembers what it has already seen, and a recurrent net does that by feeding its previous output back into itself. This page starts with the basic recurrent loop and masking, then back propagation, then the LSTM and GRU cells that fix the vanishing gradient, and ends with unsupervised LSTM, QRNN, hierarchical RNNs, and sequence analysis.
The same loop is the encoder and decoder of sequence-to-sequence models. The same notes are in SEQ2SEQ SEQUENCE TO SEQUENCE.
RNN
Everything later on the page is built on one idea: a basic NN node with a loop, previous output is merged with current input (using tanh?), for the purpose of remembering history, for time series - to predict the next X based on the previous Y.
(What is RNN?) by Andrej Karpathy - The Unreasonable Effectiveness of Recurrent Neural Networks, basically a lot of information about RNNs and their usage cases. The shapes of input and output give the use cases:
1 to N = frame captioning
N to 1 = classification
N to N = predict frames in a movie
N\2 with time delay to N\2 = predict supply and demand
The price of the loop is that the vanishing gradient is 100 times worse. Gate networks like LSTM solves vanishing gradient, which is why the LSTM and GRU sections below exist.
Before the cells, the setup question is (how to initialize?). Benchmarking RNN networks for text - don't worry about initialization, use normalization and GRU for big networks; that benchmark is kept at the end of the page.
Experimental improvements came next, around speed. Ref - ”Simplified RNN, with pytorch implementation” - is Simple Recurrent Units for Highly Parallelizable Recurrence, changing the underlying mechanism in RNNs for the purpose of parallelizing calculation, seems to work nicely in terms of speed, not sure about state of the art results. There is a Controversy regarding said work, a post by Kyunghyun Cho: the author claims he already mentioned these ideas (QRNN) first, a year before, however it seems like his ideas have also been reviewed as incremental (PixelRNN). Its probably best to read all 3 papers in chronological order and use the most optimal solution.
To try such cells yourself, RNNCELLS - recurrent shop, enables you to build complex rnns with keras. Details on their significance are inside the link.
Real sequences have different lengths, which is what masking handles. Masking for RNNs - the ideas is simple, we want to use variable length inputs, although rnns do use that, they require a fixed size input. So masking of 1’s and 0’s will help it understand the real size or where the information is in the input. Motivation: Padded inputs are going to contribute to our loss and we dont want that. Source 1 is the question of what masking in a recurrent neural network is, and source 2 is R2RT on variable length sequences in TensorFlow RNNs.
Visual attention RNNS - Same idea as masking but on a window-based cnn. Paper is the recurrent models of visual attention paper.
BACK PROPAGATION
Training that loop still comes down to back propagation, and it is easiest to see on a small net after the RNN basics above.
A great Slide about back prop, on a simple 3 neuron network, with very easy to understand calculations. The slides walk how to update the weights step by step.
LSTM
Once back propagation is clear, the vanishing gradient explains why the LSTM cell exists, and this section covers the cell and its Keras usage. The same notes are in LTSM for time series.
The best, hands down, lstm post out there is colah's. LSTM - what is? the first reference for LSTM on the web, but you should know the background before reading.
The cell in the figure carries two states. Hidden state vs cell state - you have to understand this concept before you dive in. i.e, Hidden state is overall state of what we have seen so far. Cell state is selective memory of the past. The hidden state (h) carries the information about what an RNN cell has seen over the time and supply it to the present time such that a loss function is not just dependent upon the data it is seeing in this time instant, but also, data it has seen historically. Illustrated rnn lstm gru is linked at the end of the sequence analysis section.
With the cell understood, the variants can be compared. Paper - a comparison of many LSTMs variants and they are pretty much the same performance wise. The same Paper - comparison of lstm variants, vanilla is mostly the best, forget and output gates are the most important in terms of performance. Other conclusions in the paper..
Then comes practice in Keras, mostly from Jason Brownlee's Machine Learning Mastery. Master on unrolling RNN’s introductory post is A Gentle Introduction to RNN Unrolling, which explains how the cycles in the network graph are unrolled so you can see how information moves through time. Mastery on under/over fitting lstms - but makes sense for all types of networks. Mastery on r eturn_sequence and return_state in keras LSTM is the Difference Between Return Sequences and Return States for LSTMs in Keras, and its three conclusions are:
That return sequences return the hidden state output for each input time step.
That return state returns the hidden state output and cell state for the last input time step.
That return sequences and return state can be used at the same time.
State across batches is the next question. Mastery on understanding stateful vs stateless explains that LSTM networks maintain state, and stateful stateless for time series shows the fine-grained control over when the internal state is reset when forecasting.
Connecting an LSTM to a dense output is where time can get lost. Mastery on Mastery and seq2seq timedistributed layer is How to Use the TimeDistributed Layer in Keras. The TimeDistributed Layer - used to connect 3d inputs from lstms to dense layers, in order to utilize the time element. Otherwise it gets flattened when the connection is direct, nulling the lstm purpose. Note: nice trick that doesn't increase the dense layer structure multiplied by the number of dense neurons. It loops for each time step! I.e., The TimeDistributed achieves this trick by applying the same Dense layer (same weights) to the LSTMs outputs for one time step at a time. In this way, the output layer only needs one connection to each LSTM unit (plus one bias).
For this reason, the number of training epochs needs to be increased to account for the smaller network capacity. I doubled it from 500 to 1000 to match the first one-to-one example
The tutorial walks these problems in order:
Sequence Learning Problem
One-to-One LSTM for Sequence Prediction
Many-to-One LSTM for Sequence Prediction (without TimeDistributed)
Many-to-Many LSTM for Sequence Prediction (with TimeDistributed)
The same wrapper works around a whole CNN: Mastery on wrapping cnn-lstm with time distributed, as a whole model wrap, or on every layer in the model which is equivalent and preferred. For the shapes of sequence problems themselves, Master on visual examples is Making Predictions with Sequences, on why the order of observations must be preserved. Unread - sentiment classification of IMDB movies using Keras and LSTM, a sequence classification tutorial for inputs of varying length. Very important - how to interpret LSTM neurons in keras is YerevaNN on interpreting neurons in an LSTM network. LSTM for time-series - (jakob) single point prediction, sequence prediction and shifted-sequence prediction with code.
Stateful vs Stateless: crucial for understanding how to leverage LSTM networks. On the Keras users Google Group, A good description on what it is and how to use it. The ML mastery post is the stateful and stateless forecasting tutorial again. Philippe remy on stateful vs stateless, intuition mostly with code, but not 100% clear; that post is kept at the end of the page.
Machine Learning mastery also has A good tutorial on LSTM: for one-step univariate time series forecasting, with these important notes:
Scale to -1,1, because the internal activation in the lstm cell is tanh.
2.stateful - True, needs to reset internal states, False =stateless. Great info & results HERE, with seeding, with training resets (and not) and predicting resets (and not) - note: empirically matching the shampoo input, network config, etc.
Another explanation/tutorial about stateful lstm, should be thorough.
what is return_sequence, return_states, and how to use each one and both at the same time.
Return_sequence is needed for stacked LSTM layers.
4.stacked LSTM - each layer has represents a higher level of abstraction in TIME!
Stacking makes input shapes easy to get wrong. Keras Input shape - a good explanation about differences between input_shape, dim, and what is. Additionally about layer calculation of inputs and output based on input shape, and sequence model vs API model.
Which cell to pick is still open. A comparison of LSTM/GRU/MGU with batch normalization and various initializations, GRu/Xavier/Batch are the best and recommended for RNN; that comparison is kept at the end of the page. Benchmarking LSTM variants: - An Empirical Exploration of Recurrent Network Architectures - it looks like LSTM and GRU are competitive to mutation (i believe its only in pytorch) adding a bias to LSTM works (a bias of 1 as recommended in the paper, Learning to Forget: Continual Prediction with LSTM), but generally speaking there is no conclusive empirical evidence that says one type of network is better than the other for all tests, but the mutated networks tend to win over lstm\gru variants.
That forget-gate bias is one flag in Keras. BIAS 1 in keras - unit_forget_bias: Boolean. If True, add 1 to the bias of the forget gate at initializationSetting it to true will also force bias_initializer="zeros". This is recommended in Jozefowicz et al.
The rest are the Keras training arguments that bite in practice. Validation_split arg - The validation split variable in Keras is a value between [0..1]. Keras proportionally split your training set by the value of the variable. The first set is used for training and the 2nd set for validation after each epoch.
This is a nice helper add-on by Keras, and most other Keras examples you have seen the training and test set was passed into the fit method, after you have manually made the split. The value of having a validation set is significant and is a vital step to understand how well your model is training. Ideally on a curve you want your training accuracy to be close to your validation curve, and the moment your validation curve falls below your training curve the alarm bells should go off and your model is probably busy over-fitting.
Keras is a wonderful framework for deep learning, and there are many different ways of doing things with plenty of helpers.
Return_sequence: unclear. That question asks how to use the return_sequences option together with the TimeDistributed layer.
Variable-length input comes back here too. Sequence.pad_sequences - using maxlength it will either pad with zero if smaller than, or truncate it if bigger. Batch size is fixed up front by the backend, so Using batch size for LSTM in Keras shows how to use different batch sizes when training and predicting.
Imbalanced classes? Use class_weights, another explanation here about class_weights and sample_weights, for a huge, highly imbalanced dataset fed through fit_generator. SKlearn Formula for balanced class weights and why it works, example
Sizing the layer is last. number of units in LSTM asks what "the number of units" in the LSTM cell means, and Calculate how many params are in an LSTM layer? counts the parameters that follow from it.
Back to output shapes: Understanding timedistributed in Keras, but with focus on lstm one to one, one to many and many to many - here the timedistributed is applying a dense layer to each output neuron from the lstm, which returned_sequence = true for that purpose.
This tutorial clearly shows how to manipulate input construction, lstm output neurons and the target layer for the purpose of those three problems (1:1, 1:m, m:m).
A one-directional LSTM only sees the past, which is what BIDIRECTIONAL LSTM changes. (what is?) Wiki - The basic idea of BRNNs is to connect two hidden layers of opposite directions to the same output. By this structure, the output layer can get information from past and future states.
BRNN are especially useful when the context of the input is needed. For example, in handwriting recognition, the performance can be enhanced by knowledge of the letters located before and after the current letter.
Another explanation- It involves duplicating the first recurrent layer in the network so that there are now two layers side-by-side, then providing the input sequence as-is as input to the first layer and providing a reversed copy of the input sequence to the second.
.. It allows you to specify the merge mode, that is how the forward and backward outputs should be combined before being passed on to the next layer. The options are:
‘sum‘: The outputs are added together.
‘mul‘: The outputs are multiplied together.
‘concat‘: The outputs are concatenated together (the default), providing double the number of outputs to the next layer.
‘ave‘: The average of the outputs is taken.
The default mode is to concatenate, and this is the method often used in studies of bidirectional LSTMs.
Another simplified example asks what the advantage of the forward and backward pass is over a unidirectional LSTM, and what each is better suited for.
GRU
After the LSTM, the GRU is a lighter recurrent cell built from just two gates.
A tutorial about GRU - To solve the vanishing gradient problem of a standard RNN, GRU uses, so called, update gate and reset gate. Basically, these are two vectors which decide what information should be passed to the output. The special thing about them is that they can be trained to keep information from long ago, without washing it through time or remove information which is irrelevant to the prediction. The update gate helps the model to determine how much of the past information (from previous time steps) needs to be passed along to the future. Reset gate essentially, this gate is used from the model to decide how much of the past information to forget.
Another cell in this family is the RECURRENT WEIGHTED AVERAGE (RNN-WA). What is? (a type of cell that converges to higher accuracy faster than LSTM. It implements attention into the recurrent neural network. The keras implementation is available at https://github.com/keisuke-nakata/rwa, and the whitepaper is at https://arxiv.org/pdf/1703.01253.pdf.
UNSUPERVISED LSTM
Every cell above was trained with labels; these papers train an LSTM without them.
The first is the Paper, then paper2. paper3 is An unsupervised long short-term memory neural network for event detection in cell videos. For code, see the Reddit thread on unsupervised LSTM In keras.
QRNN
After the gated recurrent cells above, the QRNN (the idea from the controversy in the RNN section) comes back as a potential competitor to the transformer. The post that made that case is kept at the end of the page.
HIERARCHICAL RNN
After the cell-level notes above, the hierarchical RNN is the one entry left without a working source. Its example was githubcode, kept at the end of the page.
NN-Sequence Analysis
Once the recurrent architectures above make predictions, the last question is why they made them.
(did not read) A causal framework for explaining the predictions of black-box sequence-to-sequence models - can this be applied to other time series prediction?
For a visual recap of the cells on this page, the Towards Data Science post animated-rnn-lstm-and-gru-ef124d06cf45 is at https://towardsdatascience.com/animated-rnn-lstm-and-gru-ef124d06cf45. The Illustrated rnn lstm gru guide, illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21, is at https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21. The GRU tutorial, understanding-gru-networks-2ef37df6c9be, is Saul Dobilas's visual explanation of Gated Recurrent Units with an end to end Python example, at https://towardsdatascience.com/understanding-gru-networks-2ef37df6c9be.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
- Towards Data Science: qrnn-a-potential-competitor-to-the-transformer-86b5aef6c137. This address no longer opens: https://towardsdatascience.com/qrnn-a-potential-competitor-to-the-transformer-86b5aef6c137 - Benchmarking RNN networks for text. This address no longer opens: https://danijar.com/benchmarking-recurrent-networks-for-language-modeling - Philippe remy. This address no longer opens: http://philipperemy.github.io/keras-stateful-lstm/ - comparison. This address no longer opens: https://danijar.com/language-modeling-with-layer-norm-and-gru/ - Jozefowicz et al.. This address no longer opens: http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf - githubcode. This address no longer opens: https://github.com/keras-team/keras/blob/master/examples/mnist_hierarchical_rnn.py
Last updated
Was this helpful?