Preparing External Market Data for QTPyLib Backtests

# Data Workflow


<img alt="QTPyLib" align="right" src="http://qtpylib.io/docs/latest/_images/qtpy-logo.png" style="margin:10px 10px 0;width:250px">


# Data Workflow


QTPyLib: Quantitative Trading Python Library<br>
https://github.com/ranaroussi/qtpylib

Copyright 2016 Ran Aroussi


---
<code>Licensed under the GNU Lesser General Public License, v3.0 (the "License"); You may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.gnu.org/licenses/lgpl-3.0.en.html

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
</code>

---

# Overview

This Jupyter Notebook aims to provide developers with a quick overview on the new `Workflow` module, introduced in version `1.5.47a` or QTPyLib.

The `Workflow` module includes some handly methods for working with external data sources when backtesting, and is planned to include methods for analyzing your backtest results in the near future.

\* Please refer to the [Workflow API Documentation](http://qtpylib.io/docs/latest/api.html#workflow-api) for complete list of available methods and parametrs.

---

Let's start by checking out QTPyLib's version:

```python
import qtpylib
qtpylib.__version__
```

Next, we need to import the `Workflow` module:

```python
from qtpylib import workflow as wf
```

## 1. Working with External Data

Sometimes, you want to backtest your strategies using market data you already have from sources other than the Blotter. Before you can use market data from external data sources, you'll need to convert it into a QTPyLib-compatible data format.

Once the data is converted, it can be read by your strategies as CSV files. You can also save the converted data in your Blotter's MySQL database so it can be used just like any other data captured by your Blotter.

---

### 1.1. Downloading Data

The `Workflow` module includes 4 methods for downloading market data from either Yahoo! Finance, Google or Interactive Brokers. The methods are:

- `get_data_yahoo()` - downloads daily bars from Yahoo! finance
- `get_data_yahoo_intraday()` - downloads 1-min intraday bars from Yahoo! finance (2 weeks max)
- `get_data_google_intraday()` - downloads 1-min intraday bars from Google (3 weeks max)
- `get_data_ib()` - downloads 1s or higher bars from Interactive Brokers (requires a running Blotter, connected to TWS/IB Gateway)

---

In this example, let's first download intraday market data for the S&P E-mini Futures from Yahoo! finance, so we'll have some data to work with...

```python
# get data from Yahoo! finance
df = wf.get_data_yahoo_intraday("ESZ16.CME")
df.tail()
```

Of course, if you already have your data as CSV files, simply load them into a `pd.DataFrame` that we will convert into QTPyLib-compatible data format in the next step.

```python
# read data from existing csv file
# df = pd.read_csv("~/Desktop/existing_data.csv")
```

---

### 1.2. Convert data into a QTPyLib-compatible data format

Once you have your existing data loaded as a `pd.DataFrame`, it's time to convert it into a QTPyLib-compatible data format by using the `prepare_data()` method and passing our data and the **IB-Compatible contract tuple or string** as the first argument.


```python
# prepare data for usage by QTPyLib
ibtuple = ("ES", "FUT", "GLOBEX", "USD", 201612, 0.0, "")
qtpylib_df = wf.prepare_data(ibtuple, data=df, output_path="~/Desktop/")
qtpylib_df.tail()
```

The resulting CSV file will be saved in `~/Desktop/ESZ2016_FUT.csv`

---

### 1.3. Storing Converted Data in MySQL

While this step is optional, you may want to load your converted data into QTPyLib's MySQL for future use by your strategies. **This step isn't required as you can backtest directly off of your converted CSV files.**

With your `Blotter` running and connected to TWS/IB Gateway, run this command:

```python
# store data
wf.store_data(qtpylib_df, kind="BAR")
```

---

### 1.4. Using CSV files when Backtesting

Once you have your CSV files in a QTPyLib-compatible format, you can backtest using this data using the `--data` flag when running your backtests.

Example:

```
$ python strategy.py --backtest --start 2016-12-01 --end 2016-12-31 --data ~/Desktop/ -output ~/portfolio.pkl
```

---

Please refer to [Back-Testing Using QTPyLib](http://qtpylib.io/docs/latest/algo.html#back-testing-using-qtpylib) for more information about back-testing using QTPyLib.