Yahoo Finance Historical Data Changes and Their Effect on Forex Scraping

# Does Yahoo/Google no longer support web-scraping of FOREX data?


# Does Yahoo/Google no longer support web-scraping of FOREX data?












I use Python 3.6 to develop quantitative FOREX trading strategies. I used to use Yahoo or Google in conjunction with `pandas.datareader` in order to collect forex data for day to day strategy development. However, lately, these the `pandas` API hasn't been working at all. I'm not sure if they changed the way FOREX data is accessed... Using something like `EURUSD=X' or`EUR=X`usually works as an argument for the 'name' of the security.

I've also tried using `ffn` for python. It is a package that makes downloading data/backtesting very simple. However, I've found that even this has stopped functioning.

Did yahoo and google change their data policies surrounding FOREX data? Or am I doing something completely wrong?

The datareader and ffn are both trying to access this URL to no avail: Yahoo: http://ichart.finance.yahoo.com/table.csv

## Answer by Brian from QuantRocket (score 6, accepted)

https://quant.stackexchange.com/a/34260

Yahoo has changed their site structure. The new download URLs look like this:

https://query1.finance.yahoo.com/v7/finance/download/MSFT?period1=1463461200&period2=1494910800&interval=1d&events=history&crumb=lHxk.yfuuzZ

These links originate from pages like this: https://finance.yahoo.com/quote/MSFT/history?period1=1463461200&period2=1494910800&interval=1d&filter=history&frequency=1d

They've switched to Unix timestamps for the date format, and the download URL now validates a cookie (the `crumb` param in the query string), which means that anyone who wants to scrape the data will need to deal with cookies.

There's a fuller discussion at Elite Trader: https://www.elitetrader.com/et/threads/yahoo-historical-data-did-they-change-the-url-recently.309554/

## Answer by siegel (score 2)

https://quant.stackexchange.com/a/34253

They are trying to make this more difficult. In matlab, you can bypass it by pretending to be a chrome user using weboptions, and pretending to access from Chrome:

https://www.mathworks.com/help/matlab/ref/weboptions.html

```
options.UserAgent = 'Chrome';
```

Maybe something similar is possible in python?

## Answer by Denis Alaev (score 0)

https://quant.stackexchange.com/a/34751

It seems that after several changes Yahoo Finance closed their API forever. The API was closed on May 15, one month ago already.

I've tested several alternatives and found that https://eodhistoricaldata.com the best one for those who used Yahoo Finance. They provide raw data, adjusted closes and splits/dividends.

They also have CSV output, with very similar format for Yahoo Finance users. Also there is a https://intrinio.com/ data provider, looks good, but they much more expensive, have no data for Mutual Funds and API is very different in compare to Yahoo Finance. Then you need to significantly change your code.