Figure 4

The memory cell of a LSTM network. The three gates (dark-gray nodes) collect activations from both inside and outside the block and control the cell via multiplicative units (small white nodes). The (white) node in the center of the cell is the constant error carousel (CEC), which ensures the constant error flow needed for learning long time dependencies.