Why Fama–French and Wilshire Market Returns Can Differ
Summary
The document compares the Fama–French market return with the Wilshire 5000 as market proxies in a three-factor regression of mutual fund returns. The accepted answer describes the Fama–French market portfolio as a value-weighted return on eligible U.S.-incorporated ordinary common shares listed on the NYSE, AMEX, or NASDAQ, and notes that the excess market factor must be combined with the risk-free rate to recover the total market return. It also points to share codes and prior-month market capitalization as relevant construction details.
As a diagnostic, the answer reports a near-perfect correlation between a Wilshire 5000 index fund and the Fama–French series over the stated sample, raising the possibility of a data alignment or distribution-handling problem in the question’s lower reported correlation. A second answer mentions possible sample-screen differences, including financial firms, return-history requirements, and positive book equity. These are suggested explanations rather than a definitive reconciliation; the response’s illustrative reconstruction is setup-specific, and historical methodology details may require checking.
Key ideas
- The Fama–French market return is built from eligible U.S. common shares across major U.S. exchanges and weighted by market value.
- The published market factor is an excess return, so the risk-free rate must be added to obtain the total market return.
- A large correlation gap between market proxies can warrant checking date alignment and whether distributions are included.
- Differences in eligible securities, sample screens, and weighting inputs can affect market proxy comparisons.
- The document presents possible sample differences but does not establish which one explains the regression results.
Tags
Full text
# Fama and French (market premium) factor
# Fama and French (market premium) factor
Currently I am using the Fama and French 3 factor model to explain the performance of mutual funds using monthly returns from 2000 to 2017.
I use two market proxies: (1) RM-RF, obtained directly from Kenneth French website.
(2) Wilshire 5000-RF, obtained from Bloomberg Terminal.
RF is the 1 month U.S. t-bill.
For some reason when I run my regressions, the R-squared using (2) Wilshire 5000 as a market proxy is higher than when (1) RM was used as a market proxy.
The correlation coefficient between the two proxies is 0.93, which suggest that there differences (probably in the composition of the Fama and French market proxy).
I was just wondering if there are any technical differences between the two market proxies?
If RM is a value weighted of all investable equities in the U.S., the correlation coefficient should be much higher than .93 relative to Wilshire 5000.
Thanks.
## Answer by Matthew Gunn (score 3, accepted)
https://quant.stackexchange.com/a/35567
### Update
I downloaded the return series for WFIVX (a Wilshire 5000 index fund) and I calculate a correlation coefficient of .9991 with the Fama-French market return series from Ken French's website (for 2000 to 2017)! So I think something is wrong with your .93 calculation?
Are your monthly returns over the same period (i.e. end of month $t-1$ to end of month $t$)). Do they include all distributions etc...?
### How Fama and French calculate the market return (RM)
From Ken French's webpage:
> Rm-Rf, the excess return on the market, value-weight return of all CRSP firms incorporated in the US and listed on the NYSE, AMEX, or NASDAQ that have a CRSP share code of 10 or 11 at the beginning of month t, good shares and price data at the beginning of t, and good return data for t minus the one-month Treasury bill rate (from Ibbotson Associates).
That the first digit of the CRSP share code is 1 means that the shares are ordinary common shares (and not certificates or ADRs etc...)
The CRSP coding for the 2nd digit are:
```
Code Definition
0 Securities which have not been further defined.
1 Securities which need not be further defined.
2 Companies incorporated outside the US
3 Americus Trust Components (Primes and Scores).
4 Closed-end funds.
5 Closed-end fund companies incorporated outside the US
8 REIT's (Real Estate Investment Trusts).
```
#### Does this match Fama-French market return?
Basically yes.
The below SQL code achieves a mean 0.4 basis point absolute different with the Fama-French `rm` factor (from their website) since 2000.
It gets a bit more dodgy in the early 1980s and I'd have to check all the details a bit better, but the big point is that you can about all the way there by:
- Only use NYSE, AMEX, and NASDAQ stocks.
- Only use regular, common shares (and no REITs)
I've included some SQL code below. (It runs on my idiosyncratic setup, and I'm including just for illustrative purposes... you won't be able to run this.):
```
SELECT t1.date, SUM(t1.ret * t2.prc * t2.shrout) / SUM(t2.prc * t2.shrout) as vw_ret
FROM (
SELECT t1.permno, t1.date, t1.ret
FROM q_stock.msf t1
JOIN q_stock.mse e on t1.permno = e.permno and e.event = 'NAMES' and e.date <= t1.date and t1.date <= e.nameendt -- join with events file to get share code
WHERE (exchcd = 1 or exchcd = 2 or exchcd = 3) and t1.date > 20000000 and t1.ret is not null and (shrcd = 10 or shrcd = 11)) t1
JOIN mycrsp.yyyymm_date_link l ON t1.date = l.date
JOIN q_stock.msf t2 ON t1.permno = t2.permno and l.prev_date = t2.date -- lagged by 1 month to get market cap weights
WHERE t2.prc > 0 -- good prior month price data (FF may do more than this)
GROUP BY t1.date
ORDER BY t1.date
```
(Note that Fama French factors have RMRF and RF so to get RM you do RM = RMRF + RF.)
## Answer by phdstudent (score 2)
https://quant.stackexchange.com/a/35565
- If you look closely to Frama French 1992 you will see that Fama-French exclude financials from their sample. They do not specify whether these are excluded from the market factor or not. That might be one of the differences from the Wilshire 5000 to the Fama-French market sample.
- Another difference is that they only include stocks that have a stock returns on 24 of the preceeding 60 months. That should be another difference.
- They only include firms with positive equity book values
If you read their data section closely I am sure you will find several other things that could make results different.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.