A Proposed Fix for ONC Cluster Quality Comparisons
Summary
The document identifies a suspected comparison error in the Optimal Number of Clusters procedure associated with de Prado and Lewis's work on detecting false investment strategies. The algorithm recursively reclusters groups whose silhouette-based t-statistics fall below the mean, then decides whether to keep the revised clustering by comparing its average score with a reference value.
The proposed correction is to compare the new clustering's average t-statistic with the overall mean from the current clustering, rather than with the mean of only the below-average clusters selected for reclustering. The author argues that the latter compares a combined set of retained and reclustered clusters against a weaker subset, so the new result may appear better even when reclustering has not improved the selected groups. An example matrix and reported statistics are mentioned as illustration. The text presents a reasoned bug report, but does not provide independent validation, a full reproduction, or a confirmed upstream resolution.
Key ideas
- The ONC procedure identifies below-average clusters for recursive reclustering.
- The author argues that the acceptance test compares the revised result to an inappropriate subset baseline.
- The suggested comparison uses the original overall mean cluster t-statistic.
- The proposed change aims to make the old and new quality measures comparable.
- The example is illustrative and the document does not report independent verification or a confirmed fix.
Tags
Full text
# Bug found in Optimal Number of Clusters algorithm - from de Prado and Lewis (2018)
# Bug found in Optimal Number of Clusters algorithm - from de Prado and Lewis (2018)
I believe I have found a bug in Optimal Number of Clusters (ONC) from the paper "Detection of False Investment Strategies Using Unsupervised Learning Methods".
```
def clusterKMeansTop(corr0,maxNumClusters=10,n_init=10):
corr1,clstrs,silh=clusterKMeansBase(corr0,maxNumClusters=corr0.shape[1]-1,n_init=n_init)
clusterTstats={i:np.mean(silh[clstrs[i]])/np.std(silh[clstrs[i]]) for i in clstrs.keys()}
tStatMean=np.mean(clusterTstats.values())
redoClusters=[i for i in clusterTstats.keys() if clusterTstats[i]<tStatMean]
if len(redoClusters)<=2:
return corr1,clstrs,silh
else:
keysRedo=[];map(keysRedo.extend,[clstrs[i] for i in redoClusters])
corrTmp=corr0.loc[keysRedo,keysRedo]
meanRedoTstat=np.mean([clusterTstats[i] for i in redoClusters])
corr2,clstrs2,silh2=clusterKMeansTop(corrTmp, \
maxNumClusters=corrTmp.shape[1]-1,n_init=n_init)
# Make new outputs, if necessary
corrNew,clstrsNew,silhNew=makeNewOutputs(corr0, \
{i:clstrs[i] for i in clstrs.keys() if i not in redoClusters},clstrs2)
newTstatMean=np.mean([np.mean(silhNew[clstrsNew[i]])/np.std(silhNew[clstrsNew[i]]) \ for i in
clstrsNew.keys()])
if newTstatMean<=meanRedoTstat:
return corr1,clstrs,silh
else:
return corrNew,clstrsNew,silhNew
```
The line
```
if newTstatMean<=meanRedoTstat:
```
should be changed to:
```
if newTstatMean<=tStatMean:
```
and delete the line:
```
meanRedoTstat=np.mean([clusterTstats[i] for i in redoClusters])
```
otherwise the algorithm is comparing apples and oranges to maximize expected quality. Because `meanRedoTstat` is the expected quality of the below-average clusters while `newTstatMean` is the expected quality of above-average + the below-average re-clustered (with `kmeansBase()`) clusters. Hence `newTstatMean` is much more likely to be larger - even if reclustering under-performs current below-average clusters.
In the picture below is an example of the recursion on a 183 x 183 matrix where tstatMean=0.62 and is equal to the below clusters which are indicated with arrow. While tstatMean should be 1.07Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.