Project 9

Gradient descent algorithm finds the global minimum of the cost function In which
information travels through the neural network from input neurons to the output neurons, while the error is calculated and propagated back through the network to update the weights.
In RNNs, information from previous time points is used as input for the next time points.
the lower the gradient is, the harder it is for the network to update the weights and the longer it takes to get to the final result.

For instance, to get from x3 to x2 we multiply x3 Then, to get from x2 to x1 we again multiply x2 So, we multiply with the same exact weight multiple times, and this is where the problem arises
when you multiply something by a small number, your value decreases very quickly.
We know that weights which are assigned at the start of the neural network with the random values, which are close to zero, and multiply x1, x2, x3, … by this value, your gradient becomes less and less with each multiplication

if weight is small, you have to vanish gradient problem, and if weight is large, you have exploding gradient problem.

Solutions to the vanishing gradient problem
In case of exploding gradient
stop backpropagating after a certain point
put a maximum limit on a gradient

In case of vanishing gradient
initialize weights so that vanishing gradient is minimized