Interpreting Logistic Regression P-Values in Strategy Research

# How important is p-value in a logistic regression based strategy?


# How important is p-value in a logistic regression based strategy?












I have never really given thought to this, but training some strategies I'm testing today I noticed that my model returns an acceptable annualized return/drawdown/etc, but the model parameters are not significant (according to p-value).

To me, this tells me that these model parameters do not adequately explain the response variable's value. But if this was the case, I would expect to see very poor backtesting/walk-forward/etc results.

What context should I be viewing the p-value of parameters of a logistic regression in the context of finance?

## Answer by Dave Harris (score 0, accepted)

https://quant.stackexchange.com/a/38838

You state that you are testing models and not a model. The standard, built-in, statistical tests of significance presume you are testing only one model. The only general exception to this is using some solution such as stepwise regression which is built to allow for multiple comparisons. Procedurally, you should perform some variant of either the AIC or the BIC across your models. After this one model is selected, you would then perform a master F test. If the F-test were significant, then you would perform t-tests on each parameter. In stepwise regression, this all happens in one giant output. After you are satisfied with that, then you would perform out-of-sample validation.

If you don't do that, then your p-values don't mean anything. The AIC and BIC, and there are a few others, are actually probability statements. They are proportionate to the Bayesian posterior probability that the model is the true model under certain very stylized assumptions. You may hear the argument that this is not true for the AIC, and under certain axiomatic constructions that would be correct, but under Bayesian rather than information theoretic axioms it is just a non-normalized approximation of the Bayesian posterior.

The reason this is important is that the information criterion is the control for multiple comparisons. If you do not use this, then you would need to have a giant grand model of models with corrections for multiple comparisons.

What do you do with all of the other models? You discard them unless the information criterion is very close. Close is very small considering that these are logarithmic comparisons. Close is very very close, well under one unit apart.

Once you have chosen one model, then you look at the F-test. If it is not significant, then you discard that model too. You would then look for different variables to build models from.

If you F-test is significant then you look at your t-tests. In your case, there are no significant variables. There are several possible explanations.

The first one is that nothing is significant. There are a large number of chance correlations in finance because financial data are constructed in such a way as there cannot be independence. You can have correlations that are real, but also useless in the sense that they have no predictive value at all.

A second possibility is multicollinearity. This is a serious problem because your information criterion felt this was the best model. If it is multicollinearity then you need to figure out which variables move close together and remove as many as possible. Principal components analysis can help here. Once you have reduced your variable set, start the process all over again with that reduced set.

If you had some variables that were significant and some that were not, then you cannot drop the ones that are not significant if the subsetting models had been considered under the information criterion. For example, consider $y=\alpha+\beta_1x_1+\beta_2x_2$ This model was accepted under the information criterion and the models where either $x_1$ or $x_2$ were omitted were rejected. The information criterion implies the data is necessary for a good model, but you lack sufficient power to falsify the null. It may be the case that $\beta_2$ is not zero but is very close to zero.

After you have completed all of this, you can start with your out-of-sample validation. If you do not like the validation results, start over with new types of data.

The key element of this is the information criterion. They are non-Frequentist estimators and so are not bound to the null hypothesis problem. Bayesian probability tests do not reject a null. There is no special hypothesis in Bayesian thinking. Bayesian hypotheses test if it is true and do not test whether a null is false. As a consequence, you should have as many hypotheses as you have options available.

Just so you have a feel for the differences, I am attaching a link from MIT's OpenCourseWare. Differences between types of inference. There is also a decent paper by Wagenmakers comparing t-tests with Bayesian posterior densities over 855 published t-tests in the existing literature. It is at Wagenmakers website. You should really read it to get a better feel for what a t-test is and is not.

## Answer by vonjd (score 2)

https://quant.stackexchange.com/a/38793

Basically there are at least two questions you have to ask yourself before believing "good backtests":

- Is there an economic or behavioural explanation why this should work (best would be to start from there in the first place).

- How many things did I try to arrive at this "good backtest".

The following paper is quite helpful to learn some of the basics of this kind of thinking:

Bailey, David H. and Borwein, Jonathan and Lopez de Prado, Marcos and Zhu, Qiji Jim, Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (April 1, 2014). Notices of the American Mathematical Society, 61(5), May 2014, pp.458-471. Available at SSRN: https://ssrn.com/abstract=2308659 or http://dx.doi.org/10.2139/ssrn.2308659

Abstract We prove that high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations, a practice we denote “backtest overfitting”. The higher the number of configurations tried, the greater is the probability that the backtest is overfit. Because most financial analysts and academics rarely report the number of configurations tried for a given backtest, investors cannot evaluate the degree of overfitting in most investment proposals.

The implication is that investors can be easily misled into allocating capital to strategies that appear to be mathematically sound and empirically supported by an outstanding backtest. Under memory effects, backtest overfitting leads to negative expected returns out-of-sample, rather than zero performance. This may be one of several reasons why so many quantitative funds appear to fail.