Downloading and Reading iShares ETF Holdings Data
Summary
The document demonstrates how to retrieve an iShares ETF holdings file directly from a data endpoint and load it into a dataframe, either by saving the file first or by reading it from the endpoint. The example skips introductory rows before displaying the tabular holdings. Its practical lesson is that a direct download can avoid scraping the visible webpage when a suitable file endpoint is available.
A later response cautions that the approach may have become outdated: the download format reportedly changed from CSV to an Excel-style file, and behavior may differ across country sites. It also notes that the downloaded file may be XML-like and not work with a standard spreadsheet reader. The examples concern a specific fund and endpoint, so file type, endpoint structure, and parsing steps may need adjustment over time.
Key ideas
- ETF holdings can be retrieved from a direct file endpoint instead of scraping the webpage.
- A downloaded holdings file can be loaded directly into a dataframe or saved locally first.
- Introductory rows may need to be skipped when parsing the example file.
- The endpoint format and download behavior can change across time and regions.
Tags
Full text
# Automatically get iShares ETF holdings # Automatically get iShares ETF holdings I heard that ETF's must publicly report their holdings all the time. I have seen that for example on the iShares website I can download the list of holdings as a csv file: https://www.ishares.com/us/products/239705/ishares-phlx-semiconductor-etf I imagine that there is a way to access these holdings for free automatically, maybe with some API? I checked the Blackrock API but on the main page, I didn't see any info for ETF's on the 'portfolio analysis' and 'search securities' tabs. I'm new to interacting with the web, so maybe my best bet would just be to google how to pull downloadables from webpages? Any thoughts? ## Answer by amdopt (score 8, accepted) https://quant.stackexchange.com/a/40610 No need to scrape the site. That should always be a last resort. The below will import the .csv file you are asking about and save it to a directory of your choice. If you don't want to specify a directory can eliminate `dir` and any references to it and the file will go straight to your working directory. I usually save data separately hence that option. ``` from urllib.request import urlretrieve import pandas as pd dir = '[Your directory of choice]' url = 'https://www.ishares.com/us/products/239705/ishares-phlx-semiconductor-etf/\ 1467271812596.ajax?fileType=csv&fileName=SOXX_holdings&dataType=fund' urlretrieve(url, dir + 'SOXX_holdings.csv') df = pd.read_csv(dir + 'SOXX_holdings.csv', skiprows=10) print(df.head()) ``` Alternate to above: importing data directly into a pandas dataframe instead of saving it locally by passing url as an argument. ``` import pandas as pd url = 'https://www.ishares.com/us/products/239705/ishares-phlx-semiconductor-etf/\ 1467271812596.ajax?fileType=csv&fileName=SOXX_holdings&dataType=fund' df = pd.read_csv(url, skiprows=10) print(df.head()) ``` Skipping the first 10 rows and printing the head is just how I wanted to view the data. Lot's of other things you can do from here. Good luck. ## Answer by user3528867 (score 3) https://quant.stackexchange.com/a/57243 I think the answer to this question is partly deprecated. The downloads have changed to .xls files. However, changing the .ajax and fileType does seem to work. However, downloading files from different countries then the us does not seem to work as probably a different .ajax file is used. Furthermore, using `pd.read_xls` does not work as the file seems to be xml related. (apparently also libreoffice cannot deal with the file) (btw: I don't have enough reputation points to make a comment)
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.