Market Data Redistribution: Licensing, Costs, and Free Sources
Summary
The document considers whether donated historical tick data can support a free public market-data service. Its main lesson is that collecting and redistributing intraday data involves legal and operational obligations: a distributor may need agreements with each market venue, end-user and device tracking, usage reports, and payment of applicable fees. Data users may also be contractually barred from redistributing data they obtained under their own licenses. These constraints can make a volunteer-run repository difficult to sustain, even when contributors are willing to share files.
The responses point to possible alternatives and examples, including a source of free historical foreign-exchange tick data, sample exchange message data useful for parser development, and scripts or tutorials for downloading available files. These are suggestions rather than a vetted, current catalog, and the document gives no assessment of data quality, coverage, or present licensing terms. Anyone building a data service would need to verify rights directly with providers and venues before redistributing records.
Key ideas
- Intraday market data redistribution can require venue agreements and substantial administrative work.
- Distributors may need to track users and devices, report usage, and remit fees to venues.
- An end-user license may prohibit donating data for redistribution.
- Some historical files and protocol samples may be available, but coverage and current terms need verification.
Tags
Full text
# Do you have historical tick data you want to donate? # Do you have historical tick data you want to donate? Do you have historical market/pricing ticket data that you would like to donate to the Open Source Trader project (OST)?? Please: upload your files! Once we gather some data, we'll do our best to help create an excellent web-service (RESTFul API) based repository of free market tick data. If we can afford the services, I'll host it in a cloud-platform (but that's still to-be-decided) No we are not going to resell your data!!! The only income the site has right now is revenue from CPC advertising, and we'll use this to fund the bandwidth and server space required to redistribute data for free later. Thanks! Upload here: http://www.opensourcetrader.com/api/free-tick-data/ P.S. Keep checking for updates, free market data of any size-able breadth or depth that is none-paid and redistributable for our project (visa via the now defunct 'opentick') has been hard to come by, so I'm soliciting for donations of data dumps!! ## Answer by Louis Marascio (score 6, accepted) https://quant.stackexchange.com/a/1960 While noble, unfortunately, this type of effort is not very practical. Mostly because market data is a major source of revenue for the market centers and is never simply given away, at least not in intraday form. A few things to consider: - Becoming a market data distributor is both costly and entails entering into agreements with each market center. If we simply consider US equities, that means completing contracts with a minimum of: NASDAQ, NYSE, BATS, and Direct Edge. This isn't a fun process. Simply purchasing data directly from these places, with no intent to re-distribute, entails tens to hundreds of pages of contract documents. - If you do become a distributor you will be required to maintain accurate records of your end users. Not only that, but most venues require you to maintain records of your end users uses of the data. For example, one end user using the data on two separate devices would be responsible for paying twice for their market data usage. You have to collect device fees from these users, build a report, send that report to the market center, and then cut a check to the market center for your end users uses. This isn't a fun process. Web sites such as Yahoo and Google pay handsomely for their right to redistribute data. Intraday data made available via API likely only represents a specific market center. For example, Yahoo, unsurprisingly, gets their intraday quotes from BATS. BATS is the easiest to work with in regards to market data (and most things, actually). If I were to contribute my market data to this effort I would be in violation of the many agreements I've signed as an end user of data. All of those agreements require me to indicate whether I intend to redistribute etc. Receiving data from folks that have no right to redistribute is a quick way to run into trouble, IMO. ## Answer by Jonathan Evans (score 3) https://quant.stackexchange.com/a/3098 Dukascopy publish free historical csv tick data but have a convoluted flash widget to access it. This website has a series of tutorial on developing a php script to download and extract all the compressed files. http://4xtutor.com/category/autotrade/market-data/ I have written my own python scripts which I will upload if anyone wants it There are samples of NASDAQ ITCH data which can be used for checking parser implementations against the latest protocol here: ftp://emi.nasdaq.com/ITCH/
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.