Reducing Neural Network Overfitting with Regularization and Dropout
Summary
This article describes techniques for reducing overfitting in feed-forward neural networks used to forecast market direction. It outlines L1 and L2 regularization, which penalize large model weights, and dropout, which randomly disables units during training. The examples also use learning-rate reduction when validation performance stalls, save the best validation model, and evaluate that model on a held-out test set.
The article applies these methods to a foreign-exchange direction prediction task and shows how to construct regularized and dropout-based models, including a version with a maximum weight constraint. It suggests reviewing training history and plotting a simple cumulative test-set PnL from thresholded predictions. The supplied material does not report numerical comparison results, and the PnL illustration assumes frictionless trading. It therefore presents implementation approaches, not evidence that these models produce a profitable live strategy.
Key ideas
- L1 and L2 regularization discourage large neural-network weights through training penalties.
- Dropout randomly removes units during training as a way to limit overfitting.
- Validation monitoring can guide learning-rate reductions and selection of a saved model.
- The examples forecast foreign-exchange direction and evaluate a selected model on test data.
- A cumulative PnL plot based on predictions is only illustrative because it omits trading frictions.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.