Choosing Information Measures and Bins for Financial Return Series
Summary
The document considers rolling Shannon entropy of intraday returns and asks how histogram bin size affects comparisons over time. Its answer suggests mutual information when the goal is to measure dependence between variables, with normalization by its maximum. For discrete analysis, bins should have statistically reliable counts; a chi-square test is offered as a way to assess discretization. For continuous distributions, it points to Kullback–Leibler divergence as a related information measure.
It also notes numerical and statistical limitations. Entropy estimates can be unstable when bins have very small probability mass, while KL divergence has difficulties when empirical distributions have different supports. The response does not prescribe a universal bin width or provide an empirical comparison of binning methods. Accordingly, the choice of measure and discretization should follow the analysis goal, and estimates require care around sparse observations and support mismatch.
Key ideas
- Shannon entropy summarizes uncertainty in a discretized return distribution, but its value depends on binning.
- Mutual information is suited to measuring dependence between variables and can be normalized by its maximum.
- Discrete bins should contain statistically reliable counts, which can be assessed with a chi-square test.
- Kullback–Leibler divergence provides a continuous-distribution comparison related to entropy.
- Sparse bin counts and mismatched empirical supports can make information estimates numerically difficult.
Tags
Full text
# Shannon's entropy for financial times-series (return)
# Shannon's entropy for financial times-series (return)
I'm looking at Shannon entropy, and generaly at ways to tell noise from signal when observing intraday returns (at the minute level for now). In python, e.g. I've implemented the fomula (sum of P(xi)*logP(xi) using a numpy histogram.
```
def rolling_entropy(window):
cx = np.histogram(window, bins)[0]
c_normalized = cx/float(np.sum(cx))
c_normalized = c_normalized[np.nonzero(c_normalized)]
h = -sum(c_normalized * np.log(c_normalized))
return h
```
The question remains: how to determine the best bin size to digitize the signal? I've assumed the bin size should probably be the tick, and that the number of bins should therefore be variable (max(return)-min(return))/tick size. However, if the number of bins is variable, can the return value be comparable from one time to another (since max and min are likely variable)?
Anyone has a view on this?
## Answer by lehalle (score 7, accepted)
https://quant.stackexchange.com/a/21689
What you need is more mutual information rather than Shannon entropy. It is dedicated to capture the influence of one variable on another (you can think about it as a non linear version of Pearson correlations). They are closely related since the mutual information $I$ between two variables $X$ and $Y$ reads: $$I(X;Y) = H(X,Y) - H(X|Y) - H(Y|X)$$ where $H$ is entropy. You have to normalize it (divide by its maximum), see wikipedia for more.
But your question is about binning. You have two solutions:
- keep it discrete (you really want to bin and you think it is a good idea). Hence you need bins to be statistically reliables: use a $\chi^2$ test (it is dedicated to capture the quality of discretization, see for instance this discussion on stats.stackexchange).
- you would like a continuous counterpart of all that: have a look at Kullback–Leibler divergence, it is made for that.
Just have a look at the KL formula between two distributions: $$D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^\infty p(x) \, \log\frac{p(x)}{q(x)} \, {\rm d}x $$ and compare to Shannon's entropy: $$H_S(X) = -\sum_{i} {\mathrm{P}(x_i) \log_b \mathrm{P}(x_i)}.$$
A last remarks:
- with entropy of discrete variables you have to face numerically bad situation with atoms of small mass (i.e. close to zero, since $\ln(0)$ is not that easy to obtain from a computer). For KL, you will have similar issues to solve when the (empirical) supports of the two distributions are not the same (have a look at the equation).
- there are papers about using such methods on financial time series, like Causality detection based on information-theoretic approaches in time series analysis.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.