When to Clean Financial Data for Risk and Portfolio Optimization

# Data: to clean or not to clean


# Data: to clean or not to clean












From risk management point of view using cleaned data (excluding or modifying extreme/outlier observations) would give less conservative measure as compared to real-world raw data. So they are more reluctant to base their conclusions on facts that are really observed.

However, portfolio managers would want to use cleaned data in order to get a more `robust and stable estimation of the distribution generating the large majority of the return data`.

As a result when they optimize their portfolio using some sort of risk measure (st.dev, CVaR, etc.) these two parties will get totally different picture.

Should having two different approaches be allowed in the first place? If yes, then how conflict situations (e.g. exceeding risk budgets) can be solved? In what cases do you think data cleaning is permissible? Sort of philosophical questions, but would love to know your thoughts.

Update

Thank you all for the answers.

@will, if data is error-free but has some outliers/extreme values, as per the below it seems to be fine to somehow reduce their impact in portfolio optimization:

Application of robust statistics to asset allocation models

> Many strategies for asset allocation involve the computation of the expected value and the covariance matrix of the returns of financial instruments. How much of each instrument to own is determined by an attempt to minimize risk — the variance of linear combinations of investments in these financial assets — subject to various constraints such as a given level of return, concentration limits, etc. The covariance matrix contains many parameters to estimate and two main problems arise. First, the data will very likely have outliers that will seriously affect the covariance matrix. Second, with so many parameters to estimate, a large number of return observations are required and the nature of markets may change substantially over such a long period. In this paper we discuss using robust covariance procedures, FAST-MCD, Iterated Bivariate Winsorization and Fast 2-D Winsorization, to address the first problem and penalization methods for the second. When back-tested on market data, these methods are shown to be effective in improving portfolio performance. Robust asset allocation methods have great potential to improve risk-adjusted portfolio returns and therefore deserve further exploration in investment management research.

Robust Portfolio Construction (can't find the full version)

> Outliers in asset returns factors are a frequently occurring phenomenon across all asset classes and can have an adverse influence on the performance of mean–variance optimized (MVO) portfolios. This occurs by virtue of the unbounded influence that outliers can have on the mean returns and covariance matrix estimates (alternatively, correlations and variances estimates) that are inputs are optimizer inputs. A possible solution to the problem of such outlier sensitivity of MVO is to use robust estimates of mean returns and covariance matrices in place of the classical estimates of these quantities thereby providing robust MVO portfolios. We show that the differences occurring between classical and robust estimates for these portfolios are such as to be of considerable concern to a portfolio manager. It turns out that robust distances based on a robust covariance matrix can provide reliable identification of multidimensional outliers in both portfolio returns and the exposures matrix of a fundamental factor model, something that is not possible with one-dimensional Winsorization. Multidimensional visualization combined with clustering methods is also useful for returns outlier identification. The question of using robust and classical MVO vs. optimization-based fat-tailed skewed distribution fits and downside risk measure is briefly discussed. Some other applications of robust methods in portfolio management are described, and we point out some future research that is needed on the topic.

Outliers and Portfolio Optimization

> In this paper we study the impact of outliers on global minimum variance portfolios. From the method developed by Gomez and Maravall (1997, 2000), we detect and correct outliers in Cac40 French index and in three French stocks included in it. It appears that all financial data present outliers, some of them may be explained by economic and financial events. We calculate the conditional volatility forecast for 60, 120 and 180 business days, using GARCH (1,1) model with 440 observations. As suggested by Franses and Ghijsels (1999), we show that outliers disturb the volatility estimates. Indeed, it seems better to correct outliers before forecasting volatility than used unadjusted series. To evaluate the forecast error, we compute the MSPE and the MAPE. Finally, we examine the impact of outliers on the global variance minimum portfolio structure. The weights of each stocks are significantly different if the series are beforehand ajusted or not. Moreover, portfolio evaluation is better for adjusted data rather than for unadjusted data. Consequently, it seems important to take into account outliers in portfolio optimisation because they affect portfolio variance, weights of portfolio and portfolio evaluation.

There are some other resources, but I do not have full access. Also, I clearly recall where practitioners mentioned taking care of outliers in portfolio construction. Obviously, they were clear and open about their "manipulations" (e.g. threw away 2-3 observations from 2007-08 period). That is why I was wondering if this approach has some merits.

@noob2 - I was looking at `clean.boudt` as a potential solution for outlier detection and data cleaning.

Any further thoughts on this?

## Answer by pyCthon (score 1)

https://quant.stackexchange.com/a/34467

To expand upon what others have said in the comments especially @will's. Data "cleansing" is not just about finding outliers or extreme values or manipulating data.

The examples below are fairly common scenarios from even the largest data vendors that must also be considered for historical or ongoing analysis. All of these issues below can throw an entire model off.

- Vendor provides incorrect values on a stock price or volume. Ex: The adjusted close not taking into account splits or dividends.

- Provides a incorrect contract specifications for a corporate bond or swap. EX: The wrong coupon frequency, currency code, exc..

- Historical intra-day quotes with incorrect time stamps.

- An exchange stops accepting orders but still sends bid/ask quotes mid market.