Reconstructing an SEC Filing Index from EDGAR Data
Article Quant Q&A · Author: user900476
Summary
The document investigates how a stock-market machine-learning book’s filing index may have been assembled. Its accepted answer proposes that the author scraped or collected quarterly company filing indexes from the SEC’s EDGAR database, then combined them into a master index. A Python package is offered as an example of a tool that downloads and cleans those quarterly indexes, which can then be filtered by company identifier and filing type to locate documents.
The answer compares sample index rows with the book’s CSV and suggests that the additional columns, such as ticker, exchange, and industry classification, may have come from a separate securities dataset. These steps are an informed reconstruction, not a confirmed account of the author’s exact process; the answer explicitly leaves open whether a package or custom scraper was used. It also cautions that reproducing the enriched file may require combining filing records with another source rather than downloading a single ready-made index.
Key ideas
- EDGAR quarterly filing indexes can be collected and combined into a master index.
- The master index can be filtered by company identifier and filing type to find filings.
- The book’s CSV appears to include fields beyond the basic filing index, potentially added from securities data.
- The proposed reconstruction is plausible but does not verify the book author’s exact tools or data sources.
Tags
Full text
# What is the possible source of this filing_index.csv? # What is the possible source of this filing_index.csv? I am reading the book Machine Learning for Algorithmic Trading by Stefan Jansen. And there is a file called filing_index.csv, which contains index information of some SEC filings. It can be accessed via the following link. https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/data/sec-filings/filing_index.csv Here are a few rows from the beginning of the file ``` "CIK","COMPANY_NAME","FORM_TYPE","DATE_FILED","EDGAR_LINK","QUARTER","TICKER","SIC","EXCHANGE","HITS","YEAR" 1000180,"SANDISK CORP","10-K","2013-02-19","edgar/data/1000180/0001000180-13-000009.txt",1,"SNDK","3572","NASDAQ","3",2013 1000209,"MEDALLION FINANCIAL CORP","10-K","2013-03-13","edgar/data/1000209/0001193125-13-103504.txt",1,"TAXI","6199","NASDAQ","0",2013 1000228,"HENRY SCHEIN INC","10-K","2013-02-13","edgar/data/1000228/0001000228-13-000010.txt",1,"HSIC","5047","NASDAQ","3",2013 1000229,"CORE LABORATORIES N V","10-K","2013-02-19","edgar/data/1000229/0001000229-13-000009.txt",1,"CLB","1389","NYSE","2",2013 1000232,"KENTUCKY BANCSHARES INC KY ","10-K","2013-03-28","edgar/data/1000232/0001104659-13-025094.txt",1,"KTYB","6022","OTC","0",2013 1000298,"IMPAC MORTGAGE HOLDINGS INC","10-K","2013-03-12","edgar/data/1000298/0001047469-13-002555.txt",1,"IMH","6798","NYSE MKT","0",2013 ``` This book doesn't seem to have mentioned the source of this filing index file. As someone who is unfamiliar with SEC filings, could anyone please help me find out the source of this file? Or is it made by the book author himself? ## Answer by Pleb (score 1, accepted) https://quant.stackexchange.com/a/70076 ### This is scraped from the Edgar database using a Python package or his own web-scraping tools: I believe the author constructed the `filing_index.csv` using a similar script/package as the Python package called `python-edgar` (see here for documentation). The package constructs a master-index file with all available information from the Edgar database since a user-defined year (pre-defined to 1993). As is also expressed in the documentation, you can then get company filings via the master-index file by filtering for eg. company (CIK number) and form-type using `grep` in Python (see example from the Github documentation). For completeness, I've provided quote-snippets from the package documentation that details how to get the needed information: > Stitch quarterly files to a master file python-edgar does only one thing and does it well: getting and cleaning uncompressed quarterly index files to your computer. Use command line tools, in the spirit of unix philosophy, to stitch these index files together and create our master index file. [...] Grab filings from a specific company Now that we have downloaded the index files it becomes easy, with a bit of command line scripting, to quickly filter by company and extract URLs to the filings we want with grep. In the following example we grep by CIK (1000045), store the output in an intermediate text file, which we re-open with cat and grep again by form 10-K. Prefix the paths with https://www.sec.gov/Archives/ and you'll get the full URL. The output from the above example, looks very similar to the information in the `filing_index.csv` with less details than the authors csv-file (missing columns beyond "EDGAR_LINK"): ``` 1000045|NICHOLAS FINANCIAL INC|10-K|2015-06-15|edgar/data/1000045/0001193125-15- 223218.txt|edgar/data/1000045/0001193125-15-223218-index.html 1000045|NICHOLAS FINANCIAL INC|10-K|2016-06-14|edgar/data/1000045/0001193125-16- 620952.txt|edgar/data/1000045/0001193125-16-620952-index.html 1000045|NICHOLAS FINANCIAL INC|10-K|2017-06-14|edgar/data/1000045/0001193125-17- 203193.txt|edgar/data/1000045/0001193125-17-203193-index.html 1000045|NICHOLAS FINANCIAL INC|10-K|2018-06-27|edgar/data/1000045/0001193125-18- 205637.txt|edgar/data/1000045/0001193125-18-205637-index.html ``` The author could then convert the above output to a csv-file and possibly merge it with securities data from the CRSP database (or similar), in order to give you the last columns missing in the output. In conclusion, I do not believe you will find any easy way to download the same type of information as provided in `filing_index.csv`. This might also be the case why the author has not mentioned the source. Nevertheless, I hope this provide some insight.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.