A brief intro into recurrent neural networks.

NN08 - recurrent networks, BPTT

In this notebook:

• recurrent networks
• backpropagation through time (BPTT)
• time-series models
• sequence models

Re-current networks

<img src=”rnn.png”,width=300,height=300> Examples include: Hopfield, Boltzmann, Echo-state, Long-short term memory(LSTMs), Nonlinear AutoRegressive network with eXogenous inputs (NARX), Elman and Jordan networks etc.

Can be trained by standard backprop and variations e.g. input/ output vectors are mapped to input+context/output (possibly using a sliding window approach)…

Backprop. through time (BPTT)

Given data sequence x with target values t: (x₀,t₀),(x₁,t₁),(x₂,t₂)… Unfold a recurrent network through time (e.g. k=3 below)

<img src=”bptt.png”,width=400,height=400>

• Train unfolded net with backprop. but in order, i.e. obtaining o₀, o₁, o₂…
• s₀ is normally a vector of zeros
• Each training example is of the form (s_t-1, x_t, s_t, x_t+1, s_t+1, x_t+2, t_t+2)
• typically use online learning
• after each example, average the weights to get the same U, V, W.