Neural Network Optimizers: SGD and Adam

Article MQL5 articles

Summary

This article introduces neural network optimization as the process of updating model weights to reduce prediction loss. It compares stochastic gradient descent, which updates from individual examples, batch gradient descent, which uses the full dataset, and mini-batch gradient descent, which uses smaller subsets. The discussion describes tradeoffs in computation, update noise, memory use, stability, convergence, and learning-rate sensitivity.

It then presents optimizer variants that modify basic gradient descent, including momentum-based updates, adaptive learning rates, and Adam, which combines moment estimates with bias correction. The implementation examples add these optimizers to a neural network for regression, and the article reports a small airfoil-noise dataset experiment with training and validation loss and accuracy. Adam is recommended as a reasonable first choice, but the article stresses that optimizer performance depends on the dataset, architecture, and settings. Its examples and reported experiment do not establish that one optimizer is best for trading models.

Key ideas

SGD, batch gradient descent, and mini-batch gradient descent differ in how much training data contributes to each update.
SGD can be computationally efficient but produces noisy updates and is sensitive to the learning rate.
Mini-batches balance update stability and dataset scale, with batch size requiring tuning.
Momentum and adaptive-rate methods alter basic gradient descent, while Adam combines adaptive moments with bias correction.
The article's regression example illustrates optimizer behavior but does not establish a best choice for trading applications.

Neural Network Optimizers: SGD, Mini-Batches, Momentum, and Adam

Summary

Key ideas

Tags