Building Long Historical Index Series from Multiple Data Sources
Summary
The discussion addresses why a request for index history beginning in 1960 returned data only from 2010 onward. Its practical recommendation is to locate a source that covers the missing period, preserve that historical segment locally, and periodically append recent observations from an online provider. This combines long-run coverage with the convenience of frequently refreshed data.
The answer reports that a historical S&P 500 series was available from Yahoo Finance back to 1927, and mentions another research dataset covering at least part of the earlier history. It also suggests combining that older material with FRED data. The proposed workflow is useful for constructing a continuous research series, but the cited availability is tied to the sources and time of the original discussion. It does not explain how to reconcile overlapping observations, adjust for revisions or corporate actions, or obtain weekly, monthly, and quarterly frequencies; those questions remain unresolved in the excerpt.
Key ideas
- A single data provider may not supply the full historical span requested for an index.
- An older dataset can be saved and supplemented with periodic downloads of recent observations.
- The answer reports a long S&P 500 history from Yahoo Finance and points to research data as another source.
- Combining sources requires attention to coverage and consistency, details the discussion does not address.
- The excerpt does not explain how to resample data to weekly, monthly, or quarterly frequencies.
Tags
Full text
# Download historical index data, like SP500, using Python. History going back to at least 1960, if not further back
# Download historical index data, like SP500, using Python. History going back to at least 1960, if not further back
I want to download historical data for different indices. I am using Python for this. I used the following code.
```
import pandas_datareader.data as web
start = datetime.datetime(1960, 1, 1)
end = datetime.datetime(2020, 4, 30)
SP500 = web.DataReader(['sp500'], 'fred', start, end)
```
It gave me data from 2010 onwards.
```
SP500.head()
sp500
DATE
2010-05-03 1202.26
2010-05-04 1173.60
2010-05-05 1165.90
2010-05-06 1128.15
2010-05-07 1110.88
```
Is there a way to pull the complete history? Also is it possible to pull data at daily/weekly/monthly/quarterly frequency. It does not have to be datareader, it can be any other function that can accomplish this.
## Answer by Stéphane (score 1)
https://quant.stackexchange.com/a/53802
The solution here is very simple. At any point in time, you will be able to retrieve from the internet stock market data at a daily frequency for free, as long as (1) you do not mind a possible delay of a few hours and (2) having only the past few years of data.
So, all you need to do is to find a dataset today that spans the missing time-frame. Then, you save it somewhere from which you can always retrieve it. Then, you can periodically update this old dataset so that, in practice, you only always need to download the last few days from the internet.
EDIT (following suggestion by noob)
I checked and you can go back to 1927 on Yahoo!Finance: https://finance.yahoo.com/quote/%5EGSPC/history?period1=-1325635200&period2=1588377600&interval=1d&filter=history&frequency=1d
OLD PROPOSAL
I don't know how far back you can get, but I know that Christian Dorion offers his code and his data for an option pricing paper he just published on his website. Buried in his data files are .mat (MATLAB data files) files containing among other things the S&P500 going back at least to 1990. It's not going back to 1960, but it's a good start. Combining this with FRED, averaging 252 trading days per year, that would make up around 7560 data points. Here.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.