The following are the main problems that arise when using local gradient optimization methods in neural network frameworks:
- Getting stuck in multiple local minima or at saddle points — The loss landscape contains both plateau regions where the gradient is near zero and strongly nonlinear regions where a sudden drop can push the search too far from the optimum.
- Non-uniform parameter updates — Some parameters are updated much less frequently than others, particularly when the data contains informative but rare attributes. Giving too much weight to rare attributes can lead to overfitting.
- Undetermined learning rate — A learning rate that is too low causes slow convergence and gets stuck in local minima. A rate that is too high skips over good minima or causes divergence.
- Vanishing and exploding gradients — In networks with many successive layers, the error gradient can shrink or grow uncontrollably as weight corrections propagate from the output back to the input. This reduces learning efficiency in layers located far from the output.
Major modifications of stochastic gradient descent use heuristics to address these problems. The most common ideas are accumulating momentum along the gradient and applying weaker updates for frequent attributes. These ideas have led to algorithms such as Nesterov Accelerated Gradient, Adagrad, Momentum, RMSProp, Adadelta, Adam, and Adamax. However, none of these fully solves all of the problems listed above.
Addressing these limitations requires a different approach to building neural networks — one that solves both the inefficiency of the training algorithm and the limited ability to search for an optimal architecture.