Why Machine Learning Stock-Prediction Research Is Limited in Finance Journals

# Why are there no papers about stock prediction with machine learning in leading financial journals?


# Why are there no papers about stock prediction with machine learning in leading financial journals?












I'm writing my master's thesis about stock price prediction using machine learning methods. During my literature review, I noticed that a lot of research produced on this topic is of poor quality, published in non-finance related journals or unpublished/peer reviewed alltogether. There is no paper to be found in leading journals like journal of finance or journal of financial economics on the topic.

I'm curious as to why this is the case. Did the academic world move on, and simply accept that markets are generally efficient a long time ago? Or are the leading journals overlooking a key technique that could effectively forecast stock price?

## Answer by Bob Jansen (score 39, accepted)

https://quant.stackexchange.com/a/61761

I think you're overlooking a third explanation:

Nobody that found a successful technique to generate alpha has published it. I can think of the following causes:

- If you're an academic, why share your brilliant idea?

- These techniques require a lot of data and financial data can be expensive, researches that work at firms that have access to this data don't share their findings with the public.

- Academics did find a lot of signals already the old fashioned way.

Despite this, fancy techniques such as AAD and Reinforcement Learning are discussed publicly. These methods don't generate any alpha however.

## Answer by MoneyScience (score 33)

https://quant.stackexchange.com/a/61775

In the early 2000s I met the Quant Team at Barclays Global Investors in San Francisco and I tried to convince them to submit some of their research to the journal I was managing at the time, Quantitative Finance. This was met with some mirth and incredulity and they told me that they might consider submitting some of the work that went into v2 of their model but that actually they were working on v9. It hadn't really dawned on me until this point that the academic literature only scratches the surface of the proprietary research that is being conducted in institutions world-wide.

## Answer by Kevin (score 26)

https://quant.stackexchange.com/a/61767

### There are very many  finance papers using machine learning

- One the of the top finance journals is the Review of Financial Studies (RFS). You find 87 published and peer-reviewed papers if you look up machine learning. You find a lot more recent research (working papers) 'published' by the National Bureau of Economic Research (NBER). Looking for 'machine learning in financial economics', you find at least 221 current working papers.

- Many academics agree that the US stock market is pretty (informationally) efficient and predicting individual stock returns (in a time series sense on a, say, daily basis) is very difficult due to idiosyncratic noise, with or without machine learning. However, there are plenty of other applications where machine learning (as a statistical tool) is very useful to deal with large data sets. Li, Mai, Shen and Yan (2020, RFS) use machine learning to identify innovative corporate cultures. If I remember correctly, even ``old'' textbooks like Campbell, Lo and MacKinlay (1997) have a chapter on neural networks.

### This includes asset pricing and stock returns

- Jackwerth and Menner (2020, JFE) use ML to empirically test Ross' recovery theorem. Bianchi, Büchner and Tamoni (2021, RFS) find predictability in bond returns. Gu, Kelly and Xiu (2021, RFS) use the nonlinearity of machine learning algorithms to be measure risk premiums better than (linear) regressions. Ke, Kelly, and Xiu use textual analysis to predict stock returns based on sentiment. Following textual analysis, Baker, Bloom, Davis and Kost analyse changes in VIX and realised volatility.

### But ultimately, it's just not their job

- Top finance and economics journals do not exist to discuss the best trading strategies that exist for a few months and then vanish again. That's just not the point of financial economics. Look at cross-sectional predictability: it's very easy to come up with factors that predict returns. But the entire debate in top finance journals is about what economic fundamentals explain why and how stock returns are predicable. Essentially, you have an easier time getting into an academic journal with estimating a general equilibrium model that explains why stock returns behave the way they do rather than with fitting some model that gives you an alpha for some time (without any economic meaning).

## Answer by Pleb (score 8)

https://quant.stackexchange.com/a/61764

Adding to the answer of @BobJansen there are some additional worries with complex machine learning models (eg. Neural Networks of any kind and complex tree-based approaches) that you can encounter, within the setting of volatility forecasting (or forecasting in general).

- The mechanisms behind the results can be hard to explain: For forecasting purposes, there is a genuine "disapproval" of complex machine learning models, since you have a very limited understanding of "what is going on under the hood" and thus why many of these models are considered black-box algorithms. It is easier to interpret and understand what is going on in a GARCH model (this is just an example). Due to the transparency of the GARCH model, we can summarize the fallacies of said model and improve it, in order to incorporate more stylized features found in the market (eg. leverage effect). Complex machine learning models are, in general, hard to "improve" since you do not have the same transparency as with the GARCH model. Understanding your models weaknesses is also key to understanding when it will eventually break down.

- The running cost can increase substantially: Not only that, but in order to improve over the more classical models (eg. GARCH, HAR etc.) they need more data, which drives the cost up for companies that need new data-sources (which can be costly) only to get a marginal overall forecast improvement. This questions whether the marginal improvement (in any case) is worth the increased cost for licensing new data-sources.

- Parsimony?: When you're constructing a forecasting model (of any type) you want to adhere to the principle of parsimony: simple models with great explanatory and predictable power. This principle is not really satisfied whenever your machine learning model "functions" as a black-box algorithm.

There might be lesser complex models that circumvent all of the above defined worries and still have great forecasting performance. These are just things that are nice to be aware of.

## Answer by AKdemy (score 7)

https://quant.stackexchange.com/a/63608

Was debating if I should even comment on this but then thought tonight I'm gonna have myself a real good time.

JPMorgan Machine Learning in Financial Markets Conference, Paris 2019 offers a superficial summary.

You simply do not have the data IMHO. To capture complex relationships you tend to have more parameters, which in turn leads to even more data requirements. The more time you spend with financial data, the more you realize it's remarkably noisy. On top of that, algorithms can only predict things consistent with what they have seen before. Not my answer - but here is an excellent summary.

Also, bear in mind, stock prediction is a very broad concept. Do you talk about someone who barely holds a stock for seconds? Or do you want to predict stock prices tomorrow, in 3 months or 10 years?

Seems most answers focus on the idea that someone with a profitable "model" would not want to share that with the public. Speaking of data, empirically, it seems to me that unicorns, planet 9 and primordial black holes and these folks have something in common. People (with unicorns it's mostly kids, with alpha less so, planets and black holes seem to fascinate adults and kids alike) tend to always talk about them, but no one has actually ever seen one for real.

Of course, there is Rentec, but generally, that also only applies to their Medallion Fund as their open funds frequently take a beating. Tradebot, and others, but there are not many who do something like this. This question goes hand in hand with the aforementioned summary which claims that many price changes are driven by news such as a plane crashing, terrorist attacks, virus outbreaks, tweets of Elon Musk or POTUS, merger announcements, and the like. Learning any pattern from these one-off events is likely more misleading than helpful.

Realistically, the outcome looks more like so. Spoiler alert, good luck for finding periods or strategies that outperform the S&P500. In the words of Warren Buffett, "I have talked to huge pension funds, and I have taken them through the math, and when I leave, they go out and hire a bunch of consultants and pay them a lot of money. Just unbelievable."

S&P Dow Jones Indices has good data. The Canadian example is an extreme outlier. As of Dec 31, 2020, 98.63% of funds underperformed the S&P/TSX Composite. However, US data does not look particularly good either. Pun intended, it’s bad and it’s getting worse. Maybe due to the emergence of ML? For readers who don't speak German, that article claims HIV may be connected to catalytic converters. Obviously complete rubbish but when you stare into the abyss, the abyss stares back (correlation does not imply causality; spurious, confounding, you name it).

It's difficult to find evidence against EMH and finding some "unicorns" is not sufficient evidence. I suspect they may be able to pull this off, without pure luck. Although statistically, the latter cannot be ruled out entirely, given the number of people who try to outsmart the average Joe. However, looking at these firms, they either employ technology to outrun competition (I know nothing about F1, but Bahrain last year showed the tools matter more than the craftsman) or some of the smartest people on earth, who do things intellectually, who I could not even dream about. Frequently, one goes hand in hand with the other. So to say the DC Universe of finance.

Nick Patterson gives a good overview about what they did at Rentec, which is mainly simple regression in his opinion (the whole podcast starts at 16:40, Rentec starts at 29:55 - a sentence before that is helpful).

This post has a good summary of some "basic" ideas to master for HFT. I think that is the only domain where you can consistently win but the users are like Lewis Hamilton. The best in their field, with the finest machinery on earth. Mainly, because you have a lot more data and the potential to exploit technological superiority. Compare FX tick data with +1 million quotes on any given day for liquid currency pairs vs 10 years of daily data. If you are in it for the long run, these ticks will be useless. If you have access to low latency of 20,000+ orders per second per single FIX, which is scalable to increase throughput and sub-millisecond roundtrip latency, the game will be different. InfoReach's website seems to be designed to Buffett's liking, no resources wasted to look pretty (although he would never need them).

The rest of us mortals lurks in the background and watches with disbelieve. Kenneth Rogoff and Richard Meese received an incredulous reaction to their now-famous paper showing that random-walk (RW) forecasts outperform economic models of exchange rates. Reactions were along the line of “You just cannot possibly have done it right” or "the results are obviously garbage". Rogoff makes an interesting point. After all, if money supplies are hard to predict, then one should not blame the models if exchange rates are hard to predict. Again unforeseen news that matters. Let's ignore that their finding was even more extreme. They tested predicting the exchange rate in one year, given the information about what money supplies, interest rates, and outputs are going to be in one year. However, even in this case, no economic model beat(s) the RW.

That paper was at a time when Warren Buffett frequently dwarfed the S&P500 returns. However, nowadays, the supply of capital chasing performance has dramatically expanded. Buffett no longer provides skyrocketing returns according to the usual first page of the annual letter to Berkshire shareholders comparing the annual performance of Berkshire against that of the S&P 500.

So to sum up, I think it may work for HFT, but the users are like Lewis Hamilton. The best in their field, with the finest machinery on earth. Overfitting noisy and irrelevant past data will not usually help much. I know many people will disagree, but empirical evidence tends to support my claims. Now is this all bad news? Not really. If an underlying follows a random walk, for example, Black Scholes and all its "derivatives" will likely be more useful. Probably also a reason why it is still used for pretty much everything (at least somewhere under the hood).

PS apologies for all the references, just had a real good time.

## Answer by statman (score 5)

https://quant.stackexchange.com/a/61801

Because it would not work. There was a very old paper several decades ago (dont recall but think it was in an IEEE journal on information theory) that showed that the stock market is RANDOM within an epsilon well smaller than the vig the brokers charge.

The sad truth is that all AI is genuine stupidity that assumes that correlation is causation. And those results are obfuscated by using huge fast complex programs manipulating large questionable piles of data that is too dirty and incomplete to be truly useful for such an application. Worse is that most assume a 'normal' distribution which is a logical fallacy. Unfortunately most people do not know enough about math and statistics to realize that some sort of beta distribution might be correct but that a normal distribution is always wrong.

That said, there are plenty of people who do write programs to predict stock results and some are widely sold, while others are sold to members of specific groups. But since none of them work right as shown by Taleb and his results noted in his book on Black Swans no reputable journal would hype them.

And as others noted above, if someone actually did have a method to predict stocks they would keep it totally secret and use it themselves.

## Answer by Nat (score 4)

https://quant.stackexchange.com/a/61793

tl;dr– People don't publish trade-secrets.

## They're trade-secrets.

Stuff like effective market-prediction algorithms tend to be trade-secrets:

> Trade secrets are a type of intellectual property that comprise formulas, practices, processes, designs, instruments, patterns, or compilations of information that have inherent economic value because they are not generally known or readily ascertainable by others, and which the owner takes reasonable measures to keep secret. —"Trade secret", Wikipedia [references omitted]

People don't tend to publish such trade-secrets for monetary reasons.

That said, academic publications often contain simpler work by both career-academics and rising professionals. When such research starts to look promising, the researchers may get offers to work in private, well-funded R&D labs where things are further developed. The original proof-of-concept research may stay in the literature while the trade-secrets are kept trade-secrets.

## Answer by DataAdventurer (score 4)

https://quant.stackexchange.com/a/65628

I'm sceptical about the story of the "silent genius" who has a fantastic quant model, generating him more money than he can spend.

The Nobel prize is waiting for you

It is highly of interest of the academic world to explain (abnormal) returns. You could win the Nobelprize for discovering and explaining returns (see Fama-French). The publicity and the arising potential asset management mandates would make richer than your "quant strategy" can generate profits.

High standards for publishing in scientific papers

The overwhelming number of 'quant' strategies do not satisfy statistical tests and standards for testing or the theoretical explaination for the returns. But not being formally right doesn't mean that the strategy can't be profitable (at least for some time).

## Answer by demully (score 3)

https://quant.stackexchange.com/a/61786

I upvote @BobJansen's points; and add one small incremental observation.

Institutional investor scepticism about quant isn't so much scepticism about quant. It's more the fear that quant describes the current regime. Which investors believe (maybe correctly, maybe not) they already understand. Unless quant can tell them how this has changed, and do so fundamentally rather than quantamentally, then investors feel they have little need to learn from quant. So they are free to ignore...

Meanwhile, the peer-review process for academics acts as negative click-bait for institutional investors. Institutions worry to their wit's end about relative positioning risks, versus other institutions. So the benefits of any quant alternative have to be assessed against the risk that this exposes the insto to risks that other instos do not face, because they do not have the same quant!

The equal-and-opposite problem here is that the quants who can navigate investor-space have no incentive to publish. It's maybe not their hiding any "secret sauce". It's rather the reality of their jobs requires them to be ready to flip their views on a dime... which is hard in any peer-reviewed journal. "Sorry- the regime changed, so I changed my mind, what would you do different when the facts change, Madam? (Keynes)" isn't a script that financial journals appreciate... but it happens all-too-often on trading floors...

therein lies your problem, methinks. best, DEM

## Answer by CMagnusB (score 2)

https://quant.stackexchange.com/a/63132

A few year back a publisher tried to start a serious peer reviewed publication on machine learning. In a few hour over 6000 machine learning crowd signed a letter vowing to never publish or in any other way help any publisher in the current peer review system with machine learning. I signed it too, since the field is extremely fast moving and still the results are hard to explain. Machine learning is a field where gut feeling is still a good way to be better at machine learning. And then we have those publishers that have no clue what they are doing. They might publish, but the issue is that the publisher have problem to assets who is qualified to review a paper submitted. The academic consist to approx 99% of people who have made their degree in a field where machine learning could be useful and then as a doctor of some kind have started to be interested in machine learning. 99% because there is always a black swan out there.

## Answer by achirikhin (score 1)

https://quant.stackexchange.com/a/79270

- You can't predict a random variable

- What most "working" approaches actually predict are not returns, but "expected returns" of some kind, or a functional of expected return. Of course building a trading or investment strategy need not predict even the expected return, these are the pre-ML ideas: MPT, CAPM etc.

- Nobody in their sane mind will publish anything that generates profit. The nominator. Ed Thorpe vs Black/Scholes.