Pre-trained Transformers for Stock Pricing and Factor Investing

Summary

The paper proposes a single-directional Transformer model, SERT, for pricing large-cap US stocks and applies pre-trained Transformers to stock pricing and factor investment. It compares these approaches with standard and encoder-only Transformers across pre-pandemic, pandemic, and one-year post-pandemic periods, with attention to performance during market stress. The authors report that SERT has the strongest out-of-sample R² during extreme market fluctuations, followed by pre-trained Transformers.

A trend-following strategy based on the models is presented as a way to hedge downside risk during shocks. In the pandemic-period static transaction-cost scenario, SERT's Sortino ratio is reported as 47% above buy-and-hold for an equal-weighted portfolio and 28% above it for a value-weighted portfolio. The paper also finds that a softmax signal filter does not improve strategy performance, while more attention heads yield only insignificant gains and applying layer normalization first does not help in this case. These results are specific to the study's periods and setup; the supplied text does not give transaction-cost assumptions or broader validation details.

Key ideas

• SERT is proposed for US large-cap stock pricing alongside pre-trained Transformer approaches.
• The models are compared with standard and encoder-only Transformers across three periods spanning the COVID-19 pandemic.
• SERT achieves the highest reported out-of-sample R² during extreme market fluctuations.
• The model-based trend-following strategy is presented as a hedge against downside risk during shocks.
• In the stated pandemic-period cost scenario, SERT's Sortino ratio exceeds buy-and-hold for both portfolio weighting methods.
• The tested softmax filter, extra attention heads, and layer-normalization order do not materially improve strategy performance.

Tags

# Asset Pricing in Pre-trained Transformer


# Asset Pricing in Pre-trained Transformer









This paper proposes an innovative Transformer model, Single-directional representative from Transformer (SERT), for US large capital stock pricing. It also innovatively applies the pre-trained Transformer models under the stock pricing and factor investment context. They are compared with standard Transformer models and encoder-only Transformer models in three periods covering the entire COVID-19 pandemic to examine the model adaptivity and suitability during the extreme market fluctuations. Namely, pre-COVID-19 period, COVID-19 period and 1-year post-COVID-19. The best proposed SERT model achieves the highest out-of-sample $R^2$, 11.94\% and 11.47\% respectively, when extreme market fluctuation takes place, followed by pre-trained Transformer models (11.13\% and 9.72\%). Their Trend-following-based strategy's performance also proves their excellent capability for hedging downside risks during market shocks. The proposed SERT model achieves a Sortino ratio 47\% higher than the buy-and-hold benchmark in the equal-weighted portfolio and 28\% higher in the value-weighted portfolio in the static transaction cost scenario when the pandemic period is considered. It proves that Transformer models have a strong ability to capture patterns of temporal sparsity in asset pricing factor models, especially with high volatility. I also find the softmax signal filter as the common configuration of Transformer models in alternative contexts, which only eliminates differences between models, but does not improve strategy-wise performance, while increasing attention heads improves the model performance insignificantly and applying the 'layer normalization first' method does not boost the model performance in our case.

Shown in full with attribution under the source's licence. Licence: abstract CC0

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.