Train a neural network

Training a Neural network means changing its Parameter aka the coefficients of the Matrix that compose the network to make it better at a specific task.

Do to this, we try to reduce the Loss function of the network. This is done by taking the derivative of the Loss function with respect to every Parameter so as it find the value of the parameters that minimises the loss.

Because $f(x_0 + \epsilon) = f(x_0) + \epsilon f'(x_0) + o(\epsilon)$ we know that $f$ gets smaller as $\epsilon f'(x_0)$ decreases, so using the sign of $f'$, we know how to tweak $x_0$ to reduce the value of $f$. In this case, $f$ is the loss function and $x_0$ is the current value of a parameter and $\epsilon$ is a tiny modification made to $\epsilon$.