Choosing Distribution-Distance Objectives for Sample Splits
Summary
The document considers how to choose a parameter that divides observations into two groups, with the aim of making their empirical distribution functions as different as possible. It compares maximizing the largest pointwise absolute gap between the two empirical distributions with maximizing the total area between them. The response cautions that the split rule matters: assigning lower observations to one group and higher observations to the other can mechanically maximize in-sample separation without discovering genuinely different data-generating processes.
If the goal is to identify observations generated by distinct processes, the useful framing is a classification or mixture-model problem. Under specified distributional assumptions, compare how likely each process is to have generated each observation; distributional distance and prior probabilities can inform that task. The answer names Kullback–Leibler divergence and expectation–maximization as relevant concepts, and notes that feature selection or transformation may improve separation. It does not resolve which empirical-CDF objective is preferable; the right choice depends on the inferential goal and model assumptions.
Key ideas
- Maximizing the largest empirical-CDF gap emphasizes the biggest local difference between groups.
- Maximizing the summed gaps instead rewards broad separation across the distributions.
- A split can create strong in-sample separation mechanically without revealing distinct generating processes.
- Classification requires assumptions about the candidate distributions and how likely each process is to generate observations.
- Kullback–Leibler divergence and expectation–maximization are relevant tools for comparing and estimating distributional models.
Tags
Full text
# Probability Theory: Maximizing the difference between distribution functions
# Probability Theory: Maximizing the difference between distribution functions
Given a sample of observations $X$, by changing a parameter $p$ we can divide $X$ into two subsamples $X_1$ and $X_2$ (this division is done in a non-trivial way which is nonetheless irrelevant to the problem at hand). We would like to find a value of $p$ for which subsamples $X_1$ and $X_2$ are as much as possible "distinct". To this purpose, we can consider how "far" apart the empirical distribution functions of $X_1$ and $X_2$ are. I am deciding between two objective functions:
1- $max_p \{max_x (\mid F_{X_1}(x)- F_{X_2}(x)\mid)\}$
2- $max_p \{\sum_x (\mid F_{X_1}(x) - F_{X_2}(x)\mid)\}$,
i.e., the first maximization problem searches for a value of $p$ that maximizes the biggest absolute difference between the EDFs, while the second one chooses a value of $p$ for which the area between the graphs of EDFs is maximized.
I acknowledge that the notion of distinctness might be ambiguous, however, I am looking for pros and cons of each objective functions and if there is any similar problem studied in the literature.
## Answer by krkeane (score 0)
https://quant.stackexchange.com/a/77917
The most useful thing I can suggest is to move this post to Cross Validated.
You are focusing on maximizing the difference in empirical distribution functions. If you sort your data, and assign the lower values to $X_1$ and the higher values to $X_2$, I believe your empirical distribution differences in the given sample would be maximized.
If the objective is to infer two distinct generating processes, for example a "typical" generating process $X_1 \sim N(0,1)$ and an "outlier" generating process $X_2 \sim N(0,100)$, focusing on the empirical distribution differences would not tease these two processes apart. A more typical analytic step would be to determine which distribution ($X_1$ or $X_2$) was more likely to have generated a given observation $x_i$. The statistical distance between these two parameterized generating distributions (and prior for "typical" vs "outlier" in a Bayesian framework) would determine the difficulty of the labeling task $x_i \in X_1$ or $x_i \in X_2$.
Given your model assumptions, and inference from the sample data, the distance between the parameterized generating distributions is fixed $-$ it doesn't make sense to talk about maximizing the distance between probability distributions given the sample data and labels $x_i \in X_1$ or $x_i \in X_2$.
If you are allowed to select different features, or transformations of the sample data, you may be able to find very distinct distributions ("good features" for your classification task).
A common measure of the distance between distributions is the Kullback–Leibler divergence.
The expectation–maximization algorithm ("EM algorithm") is commonly used in machine learning for similar tasks (attributing an observation to a generating distribution and estimating the parameters of the generating distributions).
## Answer by mehman (score 0)
https://quant.stackexchange.com/a/84080
I am not sure what is the goal here, but you could look into Kullback-Leibler divergence.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.