Recurrent Neural Networks | one minute summary

This is a recurring concept that you should make sure you understand

Image idea from: https://blog.floydhub.com/a-beginners-guide-on-recurrent-neural-networks-with-pytorch/

Although Recurrent Neural Networks (RNNs) are relatively old (from the 1980's), they are still the basic model underlying a lot of speech, text, audio, and financial data applications. This article is an introduction to the basic idea of RNNs.

1. Why? Data like images can often simply be processed one at a time by a feed-forward network. However sometimes data is not independent of other data, and you actually need to: 1. input multiple pieces of data together because they rely on each other (e.g. words in a sentence), or 2. the current data is influenced by past inputs (e.g. financial data over time).
2. What? A recurrent neural network (RNN) is a model that loops the information of multiple inputs back into the network in order to capture sequential or temporal data.
3. How? In RNNs, an input is split into multiple elements (a.k.a. time-steps) and fed into the model one at a time, with the output of each element being the “hidden state” that gets fed back into the model along with the next input element. The actual weights of an RNN network stay the same across all the elements of a training example, until the final output is calculated and that error used to update the weights.

Related articles:

Long Short-Term Memory (1997) | one minute summary

“Sequence to Sequence Learning with Neural Networks” (2014) | one minute summary