Choosing Chronological Data for Benford’s Law Tests
Summary
The document considers how to find data for testing whether first significant digits behave as independent, identically distributed Benford observations. It emphasizes that chronological ordering matters for the intended analysis and that the data should plausibly satisfy the independence assumption. A smooth population series is given as an unsuitable example because its leading digits would be expected to show dependence.
The response lists general-purpose sources where candidate datasets may be found, while noting that not all provide time-ordered observations. It also states the leading-digit probability rule and its extension to number bases beyond decimal. The main contribution is guidance on dataset selection and the mathematical benchmark, not an empirical Benford test. Finding a suitable dataset does not establish that its digits are IID: the data-generating process and temporal dependence still need to be examined before interpreting a goodness-of-fit result.
Key ideas
- Benford testing of chronological observations requires considering dependence between successive values.
- A smoothly evolving series may be unsuitable when its leading digits are predictably related over time.
- The leading-digit probability follows a logarithmic rule that can be expressed for different number bases.
- A dataset source alone does not show that its observations meet the IID assumption.
Tags
Full text
# How to obtain data for Benford's Law analysis?
# How to obtain data for Benford's Law analysis?
First off, let me be specific as to what I mean by "Benford's Law analysis": I'm looking to test, given some data, if the set of first significant digits are independent, identically distributed Benford random variables. As such, I would prefer the following:
- The order in which the data is presented matters e.g. the data is in chronological order.
- Reasonable expectation that the first significant digits might be IID.
Thus, for instance, doing this particular analysis on some sequence of population growth (1000 one year, 2000 the next, 4000 the year after that, etc.) would not be helpful, as certainly we would not expect those first significant digits to be independent of each other.
The only thing that comes to mind that meets the parameters I have set would be a chronologically ordered sequence of withdrawals/transactions from an account. But then how to obtain such data? I sure hope this question doesn't come across as lazy.
## Answer by amdopt (score 2)
https://quant.stackexchange.com/a/41711
Below is a short list of data sites that may be useful for this purpose. This list is not a comprehensive list by any means. Not all of the data on these sites will be chronological, but I think there are many datasets in these few sites that could be useful for testing Benford's.
http://testingbenfordslaw.com/
https://www.gapminder.org/data/
https://www.theguardian.com/data
https://www.data.gov/
https://www.kaggle.com/datasets
https://www.ncdc.noaa.gov/data-access
Also, using Google's new Dataset Search yield's a bunch of interesting tests with links to the datasets.
https://toolbox.google.com/datasetsearch
Aside from the above, the general form for using Benford's Law is:
$P(d) = log_b(d+1) - log_b(d) = log_b(1 + \frac1d)$
For other number bases $b$ where $b\geq1$ in case you wanted to test it on a dataset that is not decimal. The number set satisfies Benford's if the leading digit $d$ $(d \in \{1, ..., b-1\})$ occurs according to the general form.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.