Building and Configuring Futures Backtests in pysystemtrade

# backtesting


This is the user guide for using pysystemtrade as a backtesting platform. Before reading this you should have gone through the [introduction.](/docs/introduction.md)

Related documents:

- [Storing futures and spot FX data](/docs/data.md)
- [Using pysystemtrade as a production trading environment](/docs/production.md)
- [Connecting pysystemtrade to interactive brokers](/docs/IB.md)


This guide is divided into four parts. The first [How do I?](#how-do-i) explains how to do many common tasks. The second part [Guide](#guide) details the relevant parts of the code, and explains how to modify or create new parts. The third part [Processes](#Processes) discusses certain processes that cut across multiple parts of the code in more detail. The final part [Reference](#reference) includes lists of methods and parameters.

Table of Contents
=================


* [Table of Contents](#table-of-contents)
* [How do I?](#how-do-i)
   * [How do I.... Experiment with a single trading rule and instrument](#how-do-i-experiment-with-a-single-trading-rule-and-instrument)
   * [How do I....Create a standard futures backtest](#how-do-icreate-a-standard-futures-backtest)
   * [How do I....Create a futures backtest which estimates parameters](#how-do-icreate-a-futures-backtest-which-estimates-parameters)
   * [How do I....See intermediate results from a backtest](#how-do-isee-intermediate-results-from-a-backtest)
   * [How do I....See how profitable a backtest was](#how-do-isee-how-profitable-a-backtest-was)
   * [How do I....Change backtest parameters](#how-do-ichange-backtest-parameters)
      * [Option 1: Change the configuration file](#option-1-change-the-configuration-file)
      * [Option 2: Change the configuration object; create a new system](#option-2-change-the-configuration-object-create-a-new-system)
      * [Option 3: Change the configuration object within an existing system (not recommended - advanced)](#option-3-change-the-configuration-object-within-an-existing-system-not-recommended---advanced)
      * [Option 4: Create a private config file](#option-4-create-a-private-config-file)
      * [Option 5: Change the project defaults (definitely not recommended)](#option-5-change-the-project-defaults-definitely-not-recommended)
   * [How do I....Run a backtest on a different set of instruments](#how-do-irun-a-backtest-on-a-different-set-of-instruments)
      * [Change instruments: Change the configuration file](#change-instruments-change-the-configuration-file)
      * [Change instruments: Change the configuration object](#change-instruments-change-the-configuration-object)
   * [How do I.... run the backtest only on more recent data](#how-do-i-run-the-backtest-only-on-more-recent-data)
   * [How do I....Run a backtest on all available instruments](#how-do-irun-a-backtest-on-all-available-instruments)
   * [How do I.... Exclude some instruments from the backtest](#how-do-i-exclude-some-instruments-from-the-backtest)
   * [How do I.... Exclude some instruments from having positive instrument weights](#how-do-i-exclude-some-instruments-from-having-positive-instrument-weights)
   * [How do I....Create my own trading rule](#how-do-icreate-my-own-trading-rule)
      * [Writing the function](#writing-the-function)
      * [Adding the trading rule to a configuration](#adding-the-trading-rule-to-a-configuration)
   * [How do I....Use different data or instruments](#how-do-iuse-different-data-or-instruments)
   * [How do I... Save my work](#how-do-i-save-my-work)
* [Guide](#guide)
   * [Data](#data)
      * [Using the standard data objects](#using-the-standard-data-objects)
         * [Generic data objects](#generic-data-objects)
         * [The csvFuturesSimData object](#the-csvfuturessimdata-object)
         * [The dbFuturesSimData object](#the-dbfuturessimdata-object)
            * [Setting up MongoDB and Parquet](#setting-up-mongodb-and-parquet)
            * [Using dbFuturesSimData](#using-dbfuturessimdata)
         * [Arctic](#arctic)
            * [Backtesting with Arctic instead of Parquet](#backtesting-with-arctic-instead-of-parquet)
      * [Creating your own data objects](#creating-your-own-data-objects)
         * [The Data() class](#the-data-class)
   * [Configuration](#configuration)
      * [Creating a configuration object](#creating-a-configuration-object)
         * [1) Creating a configuration object with a dictionary](#1-creating-a-configuration-object-with-a-dictionary)
         * [2) Creating a configuration object from a file](#2-creating-a-configuration-object-from-a-file)
         * [3) Creating a configuration object from a pre-baked system](#3-creating-a-configuration-object-from-a-pre-baked-system)
         * [4) Creating a configuration object from a list](#4-creating-a-configuration-object-from-a-list)
         * [5) Creating configuration files from CSV files](#5-creating-configuration-files-from-csv-files)
      * [Project defaults and private configuration](#project-defaults-and-private-configuration)
         * [Handling defaults when you change certain functions](#handling-defaults-when-you-change-certain-functions)
         * [How the defaults and private configuration work](#how-the-defaults-and-private-configuration-work)
      * [Viewing configuration parameters](#viewing-configuration-parameters)
      * [Modifying configuration parameters](#modifying-configuration-parameters)
      * [Using configuration in a system](#using-configuration-in-a-system)
      * [Including your own configuration options](#including-your-own-configuration-options)
      * [Saving configurations](#saving-configurations)
      * [Modifying the configuration class](#modifying-the-configuration-class)
   * [System](#system)
      * [Pre-baked systems](#pre-baked-systems)
         * [Futures system for chapter 15](#futures-system-for-chapter-15)
         * [Estimated system for chapter 15](#estimated-system-for-chapter-15)
      * [Using the system object](#using-the-system-object)
         * [Accessing child stages, data, and config within a system](#accessing-child-stages-data-and-config-within-a-system)
         * [System methods](#system-methods)
      * [System Caching and pickling](#system-caching-and-pickling)
      * [Pickling and unpickling saved cache data](#pickling-and-unpickling-saved-cache-data)
      * [Advanced caching](#advanced-caching)
         * [Advanced Caching when backtesting.](#advanced-caching-when-backtesting)
         * [Advanced caching behaviour with a live trading system](#advanced-caching-behaviour-with-a-live-trading-system)
      * [Very advanced: Caching in new or modified code](#very-advanced-caching-in-new-or-modified-code)
      * [Creating a new 'pre-baked' system](#creating-a-new-pre-baked-system)
      * [Changing or making a new System class](#changing-or-making-a-new-system-class)
   * [Stages](#stages)
      * [Stage 'wiring'](#stage-wiring)
      * [Writing new stages](#writing-new-stages)
      * [Specific stages](#specific-stages)
      * [Stage: Raw data](#stage-raw-data)
         * [Using the standard RawData class](#using-the-standard-rawdata-class)
            * [Volatility calculation](#volatility-calculation)
         * [New or modified raw data classes](#new-or-modified-raw-data-classes)
      * [Stage: Rules](#stage-rules)
         * [Data and data arguments](#data-and-data-arguments)
      * [The Rules class, and specifying lists of trading rules](#the-rules-class-and-specifying-lists-of-trading-rules)
         * [Creating lists of rules from a configuration object](#creating-lists-of-rules-from-a-configuration-object)
         * [Interactively passing a list of trading rules](#interactively-passing-a-list-of-trading-rules)
         * [Creating variations on a single trading rule](#creating-variations-on-a-single-trading-rule)
         * [Using a newly created Rules() instance](#using-a-newly-created-rules-instance)
         * [Passing trading rules to a pre-baked system function](#passing-trading-rules-to-a-pre-baked-system-function)
         * [Changing the trading rules in a system on the fly (advanced)](#changing-the-trading-rules-in-a-system-on-the-fly-advanced)
      * [Stage: Forecast scale and cap](#stage-forecast-scale-and-cap)
         * [Using fixed weights](#using-fixed-weights)
         * [Calculating estimated forecasting scaling on the fly](#calculating-estimated-forecasting-scaling-on-the-fly)
            * [Pooled forecast scale estimate (default)](#pooled-forecast-scale-estimate-default)
            * [Individual instrument forecast scale estimate](#individual-instrument-forecast-scale-estimate)
      * [Stage: Forecast combine](#stage-forecast-combine)
         * [Using fixed weights and multipliers](#using-fixed-weights-and-multipliers)
         * [Using estimated weights and diversification multiplier](#using-estimated-weights-and-diversification-multiplier)
            * [Estimating the forecast weights](#estimating-the-forecast-weights)
            * [Removing expensive trading rules](#removing-expensive-trading-rules)
            * [Estimating the forecast diversification multiplier](#estimating-the-forecast-diversification-multiplier)
         * [Forecast mapping](#forecast-mapping)
      * [Stage: Position scaling](#stage-position-scaling)
         * [Using the standard PositionSizing class](#using-the-standard-positionsizing-class)
      * [Stage: Creating portfolios](#stage-creating-portfolios)
         * [Using fixed weights and instrument diversification multiplier(/systems/portfolio.py)](#using-fixed-weights-and-instrument-diversification-multipliersystemsportfoliopy)
         * [Using estimated weights and instrument diversification multiplier(/systems/portfolio.py)](#using-estimated-weights-and-instrument-diversification-multipliersystemsportfoliopy)
            * [Estimating the instrument weights](#estimating-the-instrument-weights)
            * [Using an estimated forecast diversification multiplier](#using-an-estimated-forecast-diversification-multiplier)
         * [Buffering and position inertia](#buffering-and-position-inertia)
         * [Capital correction](#capital-correction)
      * [Stage: Accounting](#stage-accounting)
         * [Using the standard Account class](#using-the-standard-account-class)
         * [accountCurve](#accountcurve)
         * [A nested accountCurveGroup](#a-nested-accountcurvegroup)
            * [Weighted and unweighted account curve groups](#weighted-and-unweighted-account-curve-groups)
         * [Testing account curves](#testing-account-curves)
         * [Costs](#costs)
* [Processes](#processes)
   * [File names](#file-names)
   * [Logging](#logging)
      * [Basic logging](#basic-logging)
      * [Advanced logging](#advanced-logging)
   * [Optimisation](#optimisation)
      * [The optimisation function, and data](#the-optimisation-function-and-data)
      * [Removing expensive assets (forecast weights only)](#removing-expensive-assets-forecast-weights-only)
      * [Pooling gross returns (forecast weights only)](#pooling-gross-returns-forecast-weights-only)
      * [Working out net costs (both instrument and forecast weights)](#working-out-net-costs-both-instrument-and-forecast-weights)
      * [Time periods](#time-periods)
      * [Moment estimation](#moment-estimation)
      * [Methods](#methods)
         * [Equal weights](#equal-weights)
         * [One period (not recommend)](#one-period-not-recommend)
         * [Bootstrapping (recommended, but slow)](#bootstrapping-recommended-but-slow)
         * [Shrinkage (okay, but tricky to calibrate)](#shrinkage-okay-but-tricky-to-calibrate)
         * [Handcrafting (recommended)](#handcrafting-recommended)
      * [Post processing](#post-processing)
   * [Estimating correlations and diversification multipliers](#estimating-correlations-and-diversification-multipliers)
   * [Specifying weights as hierarchy](#specifying-weights-as-hierarchy)
      * [Hierarchical forecast weight example](#hierarchical-forecast-weight-example)
      * [Hierarchical instrument weight example](#hierarchical-instrument-weight-example)
   * [Capital correction - varying capital](#capital-correction---varying-capital)
* [Reference](#reference)
   * [Table of standard system.data and system.stage methods](#table-of-standard-systemdata-and-systemstage-methods)
      * [Explanation of columns](#explanation-of-columns)
      * [System object](#system-object)
      * [Data object](#data-object)
      * [Raw data stage](#raw-data-stage)
      * [Trading rules stage (chapter 7 of book)](#trading-rules-stage-chapter-7-of-book)
      * [Forecast scaling and capping stage (chapter 7 of book)](#forecast-scaling-and-capping-stage-chapter-7-of-book)
      * [Combine forecasts stage (chapter 8 of book)](#combine-forecasts-stage-chapter-8-of-book)
      * [Position sizing stage (chapters 9 and 10 of book)](#position-sizing-stage-chapters-9-and-10-of-book)
      * [Portfolio stage (chapter 11 of book)](#portfolio-stage-chapter-11-of-book)
      * [Accounting stage](#accounting-stage)
   * [Configuration options](#configuration-options)
      * [Raw data](#raw-data)
         * [Calculating volatility](#calculating-volatility)
      * [Rules stage](#rules-stage)
         * [Trading rules](#trading-rules)
      * [Forecast scaling and capping stage](#forecast-scaling-and-capping-stage)
         * [Forecast scalar (fixed)](#forecast-scalar-fixed)
         * [Forecast scalar (estimated)](#forecast-scalar-estimated)
         * [Forecast cap (fixed - all classes)](#forecast-cap-fixed---all-classes)
      * [Forecast combination stage](#forecast-combination-stage)
         * [Forecast weights (fixed)](#forecast-weights-fixed)
         * [Forecast weights (estimated)](#forecast-weights-estimated)
            * [List of trading rules to get forecasts for](#list-of-trading-rules-to-get-forecasts-for)
            * [Parameters for estimating forecast weights](#parameters-for-estimating-forecast-weights)
         * [Forecast diversification multiplier (fixed)](#forecast-diversification-multiplier-fixed)
         * [Forecast diversification multiplier (estimated)](#forecast-diversification-multiplier-estimated)
            * [Forecast mapping config](#forecast-mapping-config)
      * [Position sizing stage](#position-sizing-stage)
         * [Capital scaling parameters](#capital-scaling-parameters)
      * [Portfolio combination stage](#portfolio-combination-stage)
         * [Instrument weights (fixed)](#instrument-weights-fixed)
         * [Instrument weights (estimated)](#instrument-weights-estimated)
         * [Instrument diversification multiplier (fixed)](#instrument-diversification-multiplier-fixed)
         * [Instrument diversification multiplier (estimated)](#instrument-diversification-multiplier-estimated)
         * [Buffering](#buffering)
      * [Accounting stage config](#accounting-stage-config)
         * [Buffering config](#buffering-config)
         * [Costs config](#costs-config)
         * [Capital correction config](#capital-correction-config)



# How do I?

   * [How do I.... Experiment with a single trading rule and instrument](#how-do-i-experiment-with-a-single-trading-rule-and-instrument)
   * [How do I....Create a standard futures backtest](#how-do-icreate-a-standard-futures-backtest)
   * [How do I....Create a futures backtest which estimates parameters](#how-do-icreate-a-futures-backtest-which-estimates-parameters)
   * [How do I....See intermediate results from a backtest](#how-do-isee-intermediate-results-from-a-backtest)
   * [How do I....See how profitable a backtest was](#how-do-isee-how-profitable-a-backtest-was)
   * [How do I....Change backtest parameters](#how-do-ichange-backtest-parameters)
   * [How do I....Run a backtest on a different set of instruments](#how-do-irun-a-backtest-on-a-different-set-of-instruments)
   * [How do I....Create my own trading rule](#how-do-icreate-my-own-trading-rule)
   * [How do I....Use different data or instruments](#how-do-iuse-different-data-or-instruments)
   * [How do I... Save my work](#how-do-i-save-my-work)


## How do I.... Experiment with a single trading rule and instrument

Although the project is intended mainly for working with trading systems, it's possible to do some limited experimentation without building a system. See [the introduction](introduction.md) for an example.

## How do I....Create a standard futures backtest

This creates the staunch systems trader example defined in chapter 15 of my book, using the csv data that is provided, and gives you the position in the DAX market:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system()
system.portfolio.get_notional_position("DAX")
```
See [standard futures system](#futures-system-for-chapter-15) for more.


## How do I....Create a futures backtest which estimates parameters

This creates the staunch systems trader example defined in chapter 15 of my book, using the csv data that is provided, and estimates forecast scalars, instrument and forecast weights, and instrument and forecast diversification multipliers:

```python
from systems.provided.futures_chapter15.estimatedsystem import futures_system
system=futures_system()
system.portfolio.get_notional_position("SOFR")
```

See [estimated futures system](#futures-system-for-chapter-15).



## How do I....See intermediate results from a backtest

This will give you the raw forecast (before scaling and capping) of one of the EWMAC rules for DAX futures in the standard futures backtest:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system()
system.rules.get_raw_forecast("DAX", "ewmac64_256")
```

For a complete list of possible intermediate results, use `print(system)` to see the names of each stage, and then `stage_name.methods()`. Or see [this table](#table-of-standard-systemdata-and-systemstage-methods) and look for rows marked with **D** for diagnostic. Alternatively type `system` to get a list of stages, and `system.stagename.methods()` to get a list of methods for a stage (insert the name of the stage, not stagename).


## How do I....See how profitable a backtest was

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system()
system.accounts.portfolio().stats() ## see some statistics
system.accounts.portfolio().curve().plot() ## plot an account curve
system.accounts.portfolio().percent.curve().plot() ## plot an account curve in percentage terms
system.accounts.pandl_for_instrument("US10").percent.stats() ## produce % statistics for a 10 year bond
system.accounts.pandl_for_instrument_forecast("SOFR", "carry").sharpe() ## Sharpe for a specific trading rule variation
```

For more information on what statistics are available, see the [relevant guide section](#using-the-standard-account-class).

## How do I....Change backtest parameters

The backtest looks for its configuration information in the following places:

1. Elements in the configuration object
2. If not found, in: the private YAML config if it exists here `/private/private_config.yaml`
3. If not found, in: Project defaults

Configuration objects can be loaded from [YAML](https://pyyaml.org/) files, or created with a dictionary. This suggests that you can modify the systems behaviour in any of the following ways:

1. Change or create a configuration YAML file, read it in, and create a new system
2. Change a configuration object in memory, and create a new system with it.
3. Change a configuration object within an existing system (advanced)
4. Create a private config YAML `/private/private_config.yaml` (this useful if you want to make a global change that affects all your backtest)
5. Change the project defaults (definitely not recommended)

For a list of all possible configuration options, see [this table](#configuration-options).

If you use options 2 or 3, you can [save the config](#saving-configurations) to a YAML file.

### Option 1: Change the configuration file

Configurations in this project are stored in [YAML](https://pyyaml.org) files. Don't worry if you're not familiar with YAML; it's just a nice way of creating nested dicts, lists and other Python objects in plain text. Just be aware that indentations are important, just in like Python, to create nesting.

You can make a new config file by copying this [one](/systems/provided/futures_chapter15/futuresconfig.yaml), and modifying it. Best practice is to save this as `pysystemtrade/private/this_system_name/config.yaml` (you'll need to create a couple of directories first).

You should then create a new system which points to the new config file:

```python
from sysdata.config.configdata import Config
from systems.provided.futures_chapter15.basesystem import futures_system

my_config=Config("private.this_system_name.config.yaml")
system=futures_system(config=my_config)
```

See [here](#file-names) for how to specify filenames in pysystemtrade.

### Option 2: Change the configuration object; create a new system

We can also modify a configuration object from a loaded system directly, and then create a new system with it:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system()
new_config=system.config

new_idm=1.1 ## new IDM

new_config.instrument_div_multiplier=new_idm

## Heres an example of how you'd change a nested parameter
## If the element doesn't yet exist in your config:

system.config.volatility_calculation=dict(days=20)

## If it does exist:
system.config.volatility_calculation['days']=20


system=futures_system(config=new_config)
```

This is useful if you're experimenting interactively 'on the fly'.


### Option 3: Change the configuration object within an existing system (not recommended - advanced)

If you opt for (3) you will need to understand about [system caching](#system-caching-and-pickling) and [how defaults are handled](#how-the-defaults-and-private-configuration-work). To modify the configuration object in the system directly:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system()

## Anything we do with the system may well be cached and will need to be cleared before it sees the new value...


new_idm=1.1 ## new IDM
system.config.instrument_div_multiplier=new_idm

## If we change anything that is nested, we need to change just one element to avoid clearing the defaults:
# So, do this:
system.config.volatility_calculation['days']=20

# Do NOT do this - it will wipe out all the other elements in the volatility_calculation dictionary:
# system.config.volatility_calculation=dict(days=20)


## The config is updated, but to reiterate anything that uses it will need to be cleared from the cache
```

Because we don't create a new system and have to recalculate everything from scratch, this can be useful for testing isolated changes to the system **if** you know what you're doing.

### Option 4: Create a private config file

Override default config params by putting them in a file at `/private/private_config.yaml`. This makes sense if you want to make a global change to a particular parameter rather than constantly including certain things in your configuration files. Anything in this file will overwrite the system defaults, but will in turn be overwritten by the backtest configuration YAML file. This file will also come in very handy when it comes to [using pysystemtrade as a production trading environment](/docs/production.md)


### Option 5: Change the project defaults (definitely not recommended)

I don't recommend changing the defaults - a lot of tests will fail for a start - but should you want to more information is given [here](#project-defaults-and-private-configuration).


## How do I....Run a backtest on a different set of instruments

Fixed instrument weights: You need to change the instrument weights in the configuration. Only instruments with weights have positions produced for them. Estimated instrument weights: You need to change the instruments section of the configuration.

There are two easy ways to do this - change the config file, or the config object already in the system (for more on changing config parameters see ['change backtest parameters'](#how-do-ichange-backtest-parameters) ). You also need to ensure that you have the data you need for any new instruments. See ['use my own data'](#how-do-iuse-different-data-or-instruments) below.


### Change instruments: Change the configuration file

You should make a new config file by copying this [one](/systems/provided/futures_chapter15/futuresconfig.yaml). Best practice is to save this as `pysystemtrade/private/this_system_name/config.yaml` (you'll need to create this directory).

For fixed weights, you can then change this section of the config:

```
instrument_weights:
    SOFR: 0.117
    US10: 0.117
    EUROSTX: 0.20
    V2X: 0.098
    MXP: 0.233
    CORN: 0.233
instrument_div_multiplier: 1.89
```

You may also have to change the forecast_weights, if they're instrument specific:

```
forecast_weights:
   SOFR:
     ewmac16_64: 0.21
     ewmac32_128: 0.08
     ewmac64_256: 0.21
     carry: 0.50
```


*At this stage you'd also need to recalculate the diversification multiplier (see chapter 11 of my book). See [estimating the forecast diversification multiplier](#estimating-correlations-and-diversification-multipliers).

For estimated instrument weights you'd change this section:

```
instruments: ["SOFR", "US10", "EUROSTX", "V2X", "MXP", "CORN"]
```

Note that if moving from fixed to estimated instrument weights (by changing `system.config.use_instrument_weight_estimates` to `True`), the set of instruments selected in your `system.config.instrument_weights` will be ignored; if you want to continue using this same set of instruments, you need to say so:

```python
system.config.instruments = list(system.config.instrument_weights.keys())
```

(The IDM will be re-estimated automatically)

You may also need to change this section, if you have different rules for each instrument:

```
rule_variations:
     SOFR: ['ewmac16_64','ewmac32_128', 'ewmac64_256', 'carry']
```

You should then create a new system which points to the new config file:

```python
from sysdata.config.configdata import Config

my_config=Config("private.this_system_name.config.yaml")

from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system(config=my_config)
```

See [here](#file-names) for how to specify filenames in pysystemtrade.



### Change instruments: Change the configuration object

We can also modify the configuration object in the system directly:

For fixed weights:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system()
new_config=system.config

new_weights=dict(SP500=0.5, KR10=0.5) ## create new weights
new_idm=1.1 ## new IDM

new_config.instrument_weights=new_weights
new_config.instrument_div_multiplier=new_idm

system=futures_system(config=new_config)

```

For estimated weights:

```python
from systems.provided.futures_chapter15.estimatedsystem import futures_system
system=futures_system()
new_config=system.config

new_config.instruments=["SP500", "KR10"]

del(new_config.rule_variations) ## means all instruments will use all trading rules

# this stage is optional if we want to give different instruments different sets of rules
new_config.rule_variations=dict(SP500=['ewmac16_64','carry'], KR10=['ewmac32_128', 'ewmac64_256', 'carry'])

system=futures_system(config=new_config)

```

## How do I.... run the backtest only on more recent data

You need to set the start_date in the YAML backtest configuration file:

```
## Note you must use this format
start_date: '2000-01-19'
```




## How do I....Run a backtest on all available instruments

If there are is no `instrument_weights` or `instruments` elements in the config, then the backtest will be run over all available instruments in the data. 

## How do I.... Exclude some instruments from the backtest

Refer to the [instruments document](/docs/instruments.md).

## How do I.... Exclude some instruments from having positive instrument weights

Refer to the [instruments document](/docs/instruments.md).


## How do I....Create my own trading rule

At some point you should read the relevant guide section ['rules'](#trading-rules) as there is much more to this subject than I will explain briefly here.


### Writing the function


A trading rule consists of:

- a function
- some data (specified as positional arguments)
- some optional control arguments (specified as key word arguments)


So the function must be something like these:

```python
def trading_rule_function(data1):
   ## do something with data1

def trading_rule_function(data1, arg1=default_value):
   ## do something with data1
   ## controlled by value of arg1

def trading_rule_function(data1, data2):
   ## do something with data1 and data2

def trading_rule_function(data1, data2, arg1=default_value, arg2=default_value):
   ## do something with data1
   ## controlled by value of arg1 and arg2

```
... and so on.

Functions must return a Tx1 pandas dataframe.

### Adding the trading rule to a configuration

We can either modify the YAML file or the configuration object we've already loaded into memory. See ['changing backtest parameters'](#how-do-ichange-backtest-parameters) for more details. If you want to use a YAML file you need to first save the function into a .py module, so it can be referenced by a string (we can also use this method for a config object in memory).

For example the rule imported like this:

```python
from systems.futures.rules import ewmac
```

Can also be referenced like so: `systems.futures.rules.ewmac`

Also note that the list of data for the rule will also be in the form of string references to methods in the system object. So for example to get the daily price we'd use the method `system.rawdata.daily_prices(instrument_code)` (for a list of all the data methods in a system see [stage methods](#table-of-standard-systemdata-and-systemstage-methods) or type `system.rawdata.methods()` and `system.rawdata.methods()`). In the trading rule specification this would be shown as "rawdata.daily_prices".

If no data is included, then the system will default to passing a single data item - the price of the instrument. Finally if any or all the `other_arg` keyword arguments are missing then the function will use its own defaults.

At this stage we can also remove any trading rules that we don't want. We also ought to modify the forecast scalars (See [forecast scale estimation](#calculating-estimated-forecasting-scaling-on-the-fly), forecast weights and probably the forecast diversification multiplier (see [estimating the forecast diversification multiplier](#estimating-correlations-and-diversification-multipliers)). If you're estimating weights and scalars (i.e. in the pre-baked estimated futures system provided) this will be automatic.

*If you're using fixed values (the default) then if you don't include a forecast scalar for the rule, it will use a value of 1.0. If you don't include forecast weights in your config then the system will default to equally weighting. But if you include forecast weights, but miss out the new rule, then it won't be used to calculate the combined forecast.*

Here's an example for a new variation of the EWMAC rule. This rule uses two types of data - the price (stitched for futures), and a precalculated estimate of volatility.

YAML: (example)
```
trading_rules:
  .... existing rules ...
  new_rule:
     function: systems.futures.rules.ewmac
     data:
         - "rawdata.daily_prices"
         - "rawdata.daily_returns_volatility"
     other_args:
         Lfast: 10
         Lslow: 40
#
#
## Following section is for fixed scalars, weights and div. multiplier:
#
forecast_scalars:
  ..... existing rules ....
  new_rule=10.6
#
forecast_weights:
  .... existing rules ...
  new_rule=0.10
#
forecast_div_multiplier=1.5
#
#
## Alternatively if you're estimating these quantities use this section:
#
use_forecast_weight_estimates: True
use_forecast_scale_estimates: True
use_forecast_div_mult_estimates: True

rule_variations:
     SOFR: ['ewmac16_64','ewmac32_128', 'ewmac64_256', 'new_rule']
#
# OR if all variations are the same for all instruments
#
rule_variations: ['ewmac16_64','ewmac32_128', 'ewmac64_256', 'new_rule']
#
```



Python (example - assuming we already have a config object loaded to modify)

```python

from systems.trading_rules import TradingRule

# method 1
new_rule = TradingRule(
   dict(function="systems.futures.rules.ewmac", data=["rawdata.daily_prices", "rawdata.daily_returns_volatility"],
        other_args=dict(Lfast=10, Lslow=40)))

# method 2 - good for functions created on the fly
from systems.futures.rules import ewmac

new_rule = TradingRule(dict(function=ewmac, data=["rawdata.daily_prices", "rawdata.daily_returns_volatility"],
                            other_args=dict(Lfast=10, Lslow=40)))

## both methods - modify the configuration
config.trading_rules['new_rule'] = new_rule

## If you're using fixed weights and scalars

config.forecast_scalars['new_rule'] = 7.0
config.forecast_weights = dict(...., new_rule=0.10)  ## all existing forecast weights will need to be updated
config.forecast_div_multiplier = 1.5

## If you're using estimates

config.use_forecast_scale_estimates = True
config.use_forecast_weight_estimates = True
use_forecast_div_mult_estimates: True

config.rule_variations = ['ewmac16_64', 'ewmac32_128', 'ewmac64_256', 'new_rule']
# or to specify different variations for different instruments
config.rule_variations = dict(SP500=['ewmac16_64', 'ewmac32_128', 'ewmac64_256', 'new_rule'], US10=['new_rule', ....)
```

Once we've got the new config, by which ever method, we just use it in our system, eg:

```python
## put into a new system

from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system(config=config)
```

## How do I....Use different data or instruments

The default data used for the simulation is CSV files for futures stitched prices, FX and contract related data. It's my intention to update this and try to keep it reasonably current with each release. The data is stored in the [data/futures directory](/data/futures). See [update Sep 2025](/docs/data.md#note-on-outdated-shipped-csv-data)

You can update that data, if you wish. Be careful to save it as a CSV with the right formatting, or pandas will complain. Check that a file is correctly formatted like so:

```python
import pandas as pd
test=pd.read_csv("filename.csv")
test
```
You can also add new files for new instruments. Be sure to keep the file format and header names consistent.

You can create your own directory for CSV files. For example supposed you wanted to get your adjusted prices from `pysystemtrade/private/system_name/adjusted_price_data`. Here is how you'd use it:

```python
from sysdata.sim.csv_futures_sim_data import csvFuturesSimData
from systems.provided.futures_chapter15.basesystem import futures_system

data=csvFuturesSimData(csv_data_paths=dict(csvFuturesAdjustedPricesData = "private.system_name.adjusted_price_data"))
system=futures_system(data=data)
```
Notice that we use Python style "." internal references within a project, we don't give actual path names. See [here](#file-names) for how to specify filenames in pysystemtrade.

The full list of keys that you can use in the `csv_data_paths` are:
* `csvFuturesInstrumentData` (configuration and costs)
* `csvFuturesMultiplePricesData` (prices for current, next and carry contracts)
* `csvFuturesAdjustedPricesData` (stitched back-adjusted prices)
* `csvFxPricesData` (for FX prices)
* `csvRollParametersData` (for roll configuration)
  
Note that you can't put adjusted prices and carry data in the same directory since they use the same file format.

There is more detail about using CSV files [here](#the-csvfuturessimdata-object).

If you want to store your data in MongoDB databases instead you need to [use a different data object](#the-dbfuturessimdata-object).

If you want to get data from a different place (eg a database, yahoo finance, broker, quandl...) you'll need to [create your own Data object](#creating-your-own-data-objects).

If you want to use a different set of data values (eg equity EP ratios, interest rates...) you'll need to [create your own Data object](#creating-your-own-data-objects).

If you want to delve deeper into data storage see the document [working with futures data](/docs/data.md)

## How do I... Save my work

To remain organised it's good practice to save any work into a directory like `pysystemtrade/private/this_system_name/` (you'll need to create the directory first). If you plan to contribute to GitHub, just be careful to avoid adding 'private' to your commit ([you may want to read this](https://24ways.org/2013/keeping-parts-of-your-codebase-private-on-github/)).

You can save the contents of a system cache to avoid having to redo calculations when you come to work on the system again (but you might want to read about [system caching and pickling](#system-caching-and-pickling) before you reload them).

```python
from systems.provided.futures_chapter15.basesystem import futures_system

system = futures_system()
system.accounts.portfolio().sharpe() ## does a whole bunch of calculations that will be saved in the cache

system.cache.pickle("private.this_system_name.system.pck") ## use any file extension you like

## In a new session
from systems.provided.futures_chapter15.basesystem import futures_system

system = futures_system()
system.cache.unpickle("private.this_system_name.system.pck")

## this will run much faster and reuse previous calculations
# only complex accounting p&l objects aren't saved in the cache
system.accounts.portfolio().sharpe()

```

You can also save a config object into a YAML file - see [saving configuration](#saving-configurations).


# Guide


The guide section explains in more detail how each part of the system works:

1. [Data](#data) objects
2. [Config](#configuration) objects and YAML files
3. [System](#system) objects,
4. [Stages](#stages) within a system.

Each section is split into parts that get progressively trickier; varying from using the standard objects that are supplied up to writing your own.


## Data

A data object is used to feed data into a system. Data objects work with a particular **kind** of data (normally asset class specific, eg futures) from a particular **source** (for example CSV files, databases and so on).

### Using the standard data objects

Two kinds of specific data object is currently provided with the system in the current version - `csvFuturesSimData` (CSV files) and `dbFuturesSimData` (database storage)

See [working with futures data](/docs/data.md)


#### Generic data objects

You can import and use data objects directly:

*These commands will work with all data objects - the `csvFuturesSimData` version is
used as an example.*

```python
from sysdata.sim.csv_futures_sim_data import csvFuturesSimData

data=csvFuturesSimData()

## getting data out
data.methods() ## list of methods

data.get_raw_price(instrument_code)
data[instrument_code] ## does the same thing as get_raw_price

data.get_instrument_list()
data.keys() ## also gets the instrument list

data.get_value_of_block_price_move(instrument_code)
data.get_instrument_currency(instrument_code)
data.get_fx_for_instrument(instrument_code, base_currency) # get FX rate between instrument currency and base currency

```

Or within a system:

```python
## using with a system
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system(data=data)

system.data.get_instrument_currency(instrument_code) # and so on
```

(Note that when specifying a data item within a trading [rule](#stage-rules) you
should omit the system eg `data.get_raw_price`)

If you set the start_date configuration option, then only a subset of the data will be shown:


```python
## using with a system
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system(data=data)

# We could also do this in the YAML file. Note the formatting used must be the same
system.config.start_date = '2000-01-19'

## or as a datetime (won't work in YAML obviously)
import datetime
system.config.start_date = datetime.datetime(2000,1,19)
```


#### The csvFuturesSimData object

The `csvFuturesSimData` object works like this:

```python
from sysdata.sim.csv_futures_sim_data import csvFuturesSimData

## with the default folders
data=csvFuturesSimData()

## OR with different folders, by providing a dict containing the folder(s) to use
data=csvFuturesSimData(csv_data_paths = dict(key_name = "pathtodata.with.dots"))

# Permissible key names are 'csvFxPricesData' (FX prices), 'csvFuturesMultiplePricesData' 
# (for carry and forward prices),
# 'csvFuturesAdjustedPricesData' and 'csvFuturesInstrumentData' (configuration and costs).
# If a keyname is not present then the system defaults will be used

# An example to override with FX data stored in /psystemtrade/private/data/fxdata/:

data=csvFuturesSimData(csv_data_paths = dict(csvFxPricesData="private.data.fxdata"))

# WARNING: Do not store multiple_price_data and adjusted_price_data in the same directory
#          They use the same file names!

## getting data out
data.methods() ## will list any extra methods
data.get_instrument_raw_carry_data(instrument_code) ## specific data for futures

## using with a system
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system(data=data)
system.data.get_instrument_raw_carry_data(instrument_code)
```

Each relevant pathname must contain CSV files of the following four types (where code is the instrument_code):

1. Static configuration and cost data- `instrument_config.csv` headings: Instrument, Pointsize, AssetClass, Currency. Additional headings for costs: Slippage, PerBlock, Percentage, PerTrade. See ['costs'](#costs) for more detail.
2. Roll parameters data. See [storing futures and spot FX data](/docs/data.md) for more detail.
3. Adjusted price data- `code.csv` (eg SP500.csv) headings: DATETIME, PRICE
4. Carry and forward data - `code.csv` (eg AEX.csv): headings: DATETIME, PRICE,CARRY,FORWARD,CARRY_CONTRACT PRICE_CONTRACT, FORWARD_CONTRACT
5. Currency data - `ccy1ccy2fx.csv` (eg AUDUSDfx.csv) headings: DATETIME, FXRATE

DATETIME should be something that `pandas.to_datetime` can parse. Note that the price in (2) is the continuously stitched price (see [volatility calculation](#volatility-calculation) ), whereas the price column in (3) is the price of the contract we're currently trading.

At a minimum we need to have a currency file for each instrument's currency against the default (defined as "USD"); and for the currency of the account we're trading in (i.e. for a UK investor you'd need a `GBPUSDfx.csv` file). If cross rate files are available they will be used; otherwise the USD rates will be used to work out implied cross rates.

See data in subdirectories [pysystemtrade/data/futures](/data/futures) for files you can modify:

- [adjusted prices](/data/futures/adjusted_prices_csv),
- [configuration and costs](/data/futures/csvconfig),
- [Futures specific carry and forward prices](/data/futures/multiple_prices_csv)
- [Spot FX prices](/data/futures/fx_prices_csv)

For more information see the [futures data document](/docs/data.md#csvfuturessimdata).


#### The dbFuturesSimData object

This is a simData object which gets its data from [MongoDB](https://mongodb.com) (static) and [Parquet](https://parquet.apache.org/) (time series). It is better for live trading. For production code, and storing large amounts of data (eg for individual futures contracts) we probably need something more robust than CSV files.


##### Setting up MongoDB and Parquet

Obviously you will need to make sure you already have a MongoDB instance running. You might find you already have one running, in Linux use `ps wuax | grep mongo` and then kill the relevant process. 

Because the MongoDB data isn't included in the GitHub repo, before using this you need to write the required data into Mongo and Parquet. You can do this from scratch, as per the ['futures data workflow'](/docs/data.md#part-1-a-futures-data-workflow). Alternatively you can run the following scripts which will copy the data from the existing GitHub CSV files:

- [Adjusted prices](/sysinit/futures/repocsv_adjusted_prices.py)
- [Multiple prices](/sysinit/futures/repocsv_multiple_prices.py)
- [Spot FX prices](/sysinit/futures/repocsv_spotfx_prices.py)
- [Spread data](/sysinit/futures/repocsv_spread_costs.py)

Of course it's also possible to mix these two methods.

##### Using dbFuturesSimData

Once you have the data it's just a matter of replacing the default csv data object:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
from sysdata.sim.db_futures_sim_data import dbFuturesSimData

# with the default database
data = dbFuturesSimData()

# using with a system
system = futures_system()
print(system.accounts.portfolio().sharpe())
```

#### Arctic

Early versions of this project used [Arctic](https://github.com/manahl/arctic) for storing time series data. Since November 2023, Parquet is the default. See [the reasoning behind the change](https://github.com/pst-group/pysystemtrade/discussions/466), [switchover instructions](https://github.com/pst-group/pysystemtrade/discussions/1290), and [required scheduling config changes](https://github.com/pst-group/pysystemtrade/discussions/1291).

The [original Arctic project](https://github.com/man-group/arctic) is no longer maintained - development has moved to [ArcticDB](https://github.com/man-group/ArcticDB)

Parquet and Arctic are very different things. Parquet is an efficient column-oriented data file format, while Arctic is a thin software layer that allows storage of time series data in a MongoDB database.

Although Parquet is not a database, data in this project sits behind an abstraction that treats it like one.

##### Backtesting with Arctic instead of Parquet

It is still possible to use the older method. For backtesting, modify the classes assigned in `sysdata.sim.db_futures_sim_data.py` to point to the Arctic classes instead of Parquet:

```python
use_sim_classes = {
    FX_DATA: arcticFxPricesData,
    ROLL_PARAMETERS_DATA: csvRollParametersData,
    FUTURES_INSTRUMENT_DATA: csvFuturesInstrumentData,
    FUTURES_MULTIPLE_PRICE_DATA: arcticFuturesMultiplePricesData,
    FUTURES_ADJUSTED_PRICE_DATA: arcticFuturesAdjustedPricesData,
    STORED_SPREAD_DATA: mongoSpreadCostData,
}
```

You would need to uncomment the Arctic imports too.


### Creating your own data objects

You should be familiar with the Python object orientated idiom before reading this section.

The [`simData()`](/sysdata/sim/sim_data.py) object is the base class for data used in simulations. From that we inherit data type specific classes such as those [for futures](/sysdata/sim/futures_sim_data.py) object. These in turn are inherited from for specific data sources, such as for csv files: [csvFuturesSimData()](/sysdata/sim/csv_futures_sim_data.py).

It is helpful if this naming scheme was adhered to: sourceTypeSimData. For example if we had some single equity data stored in a database we'd do `class EquitiesSimData(simData)`, and `class dbEquitiesSimData(EquitiesSimData)`.

So, you should consider whether you need a new type of data, a new source of data or both. You may also wish to extend an existing class. For example if you wished to add some fundamental data for futures you might define: `class fundamentalFuturesSimData(futuresSimData)`. You'd then need to inherit from that for a specific source.

This might seem a hassle, and it's tempting to skip and just inherit from `simData()` directly, however once your system is up and running it is very convenient to have the possibility of multiple data sources and this process ensures they keep a consistent API for a given data type.

It's worth reading the [documentation on futures data](/docs/data.md#modifying-simdata-objects) to understand how [csvFuturesSimData()](/sysdata/sim/csv_futures_sim_data.py) is constructed before modifying it or creating your own data objects.

#### The Data() class

Methods that you'll probably want to override:

- `get_raw_price` Returns Tx1 pandas data frame
- `get_instrument_list` Returns list of str
- `get_value_of_block_price_move` Returns float
- `get_raw_cost_data` Returns a dict cost data
- `get_instrument_currency`: Returns str
- `_get_fx_data(currency1, currency2)` Returns Tx1 pandas data frame of exchange rates
- 'get_rolls_per_year': returns int

You should not override `get_fx_for_instrument`, or any of the other private FX related methods. Once you've created a `_get_fx_data method`, then the methods in the `Data` base class will interact to give the correct FX rate when external objects call `get_fx_for_instrument()`; handling cross rates and working them out as needed.

Neither should you override 'daily_prices'.

Finally data methods should not do any caching. [Caching](#system-caching-and-pickling) is done within the system class.


## Configuration

Configuration (`config`) objects determine how a system behaves. Configuration objects are very simple; they have attributes which contain either parameters, or nested groups of parameters.


### Creating a configuration object

There are three main ways to create a configuration object:

1. Interactively from a dictionary
2. By pulling in a YAML file
3. From a 'pre-baked' system
4. By joining together multiple configurations in a list

#### 1) Creating a configuration object with a dictionary

```python
from sysdata.config.configdata import Config

my_config_dict=dict(optionone=1, optiontwo=dict(a=3.0, b="beta", c=["a", "b"]), optionthree=[1.0, 2.0])
my_config=Config(my_config_dict)
```

There are no restrictions on what is nested in the dictionary, but if you include arbitrary items like the above they won't be very useful!. The section on [configuration options](#configuration-options) explains what configuration options would be used by a system.

#### 2) Creating a configuration object from a file

This simple file will reproduce the useless config we get from a dictionary in the example above.

```
optionone: 1
optiontwo:
  a: 3.0
  b: "beta"
  c:
    - "a"
    - "b"
optionthree:
  - 1.0
  - 2.0
```

Note that as with Python the indentation in a YAML file shows how things are nested. If you want to learn more about YAML check [this out](https://pyyaml.org/wiki/PyYAMLDocumentation#YAMLsyntax).

```python
from sysdata.config.configdata import Config
my_config=Config("private.filename.yaml") ## assuming the file is in "pysystemtrade/private/filename.yaml"
```

See [here](#file-names) for how to specify filenames in pysystemtrade.


In theory there are no restrictions on what is nested in the dictionary (but the top level must be a dict); although it is easier to use str, float, int, lists and dicts, and the standard project code only requires those (if you're a PyYAML expert you can do other Python objects like tuples, but it won't be pretty).

You should respect the structure of the default config with respect to nesting, as otherwise [the defaults](#how-the-defaults-and-private-configuration-work) won't be properly filled in.

The section on [configuration options](#configuration-options) explains what configuration options are available.


#### 3) Creating a configuration object from a pre-baked system

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system()
new_config=system.config
```

Under the hood this is effectively getting a configuration from [`/systems/provided/futures_chapter15/futuresconfig.yaml`](/systems/provided/futures_chapter15/futuresconfig.yaml).

Configs created in this way will include all [the defaults populated](#how-the-defaults-and-private-configuration-work).


#### 4) Creating a configuration object from a list

We can also pass a list into `Config()`, where each item of the list contains a dict or filename. For example we could do this with the simple filename example above:

```python
from sysdata.config.configdata import Config

my_config_dict=dict(optionfour=1, optionfive=dict(one=1, two=2.0))
my_config=Config(["filename.yaml", my_config_dict])
```

Note that if there are overlapping keynames, then those in latter parts of the list of configs will override earlier versions.

This can be useful if, for example, we wanted to change the instrument weights 'on the fly' but keep the rest of the configuration unchanged.

#### 5) Creating configuration files from CSV files

Sometimes it is more convenient to specify certain parameters in a CSV file, then push them into a YAML file. If you want to use this method then you can use these two functions:

```python
from sysinit.configtools.csvweights_to_yaml import instr_weights_csv_to_yaml  # for instrument weights
from sysinit.configtools.csvweights_to_yaml import forecast_weights_by_instrument_csv_to_yaml  # forecast weights for each instrument
from sysinit.configtools.csvweights_to_yaml import forecast_mapping_csv_to_yaml # Forecast mapping for each instrument
```

These will create YAML files which can then be pasted into your existing configuration files.


### Project defaults and private configuration

Many (but not all) configuration parameters have defaults which are used by the system if the parameters are not in the object. These can be found in the [defaults.yaml file](/sysdata/config/defaults.yaml). The section on [configuration options](#configuration-options) explains what the defaults are, and where they are used.

I recommend that you do not change these defaults. It's better to use the settings you want in each system configuration file, or use a private configuration file if this is something you want to apply to all your backtests.

If this file exists, `/private/private_config.yaml`, it will be used as a private configuration file. Another option is to [put your private config in a directory external to the project](production.md#custom-private-directory).

Basically, whenever a configuration object is added to a system, if there is a private config file then we add the elements from that. Then for any remaining missing elements we add the elements from `defaults.yaml`.


#### Handling defaults when you change certain functions

In certain places you can change the function used to do a particular calculation, eg volatility estimation (This does *not* include trading rules - the way we change the functions for these is quite different). This is straightforward if you're going to use the same arguments as the original argument. However if you change the arguments you'll need to change the project `defaults.yaml` file. I recommend keeping the original parameters, and adding new ones with different names, to avoid accidentally breaking the system.


#### How the defaults and private configuration work


When added to a system the config class fills in parameters that are missing from the original config object, but are present in (i) the private YAML file and (ii) the default YAML file. For example if forecast_scalar is missing from the config, then the default value of 1.0 will be used. This works in a similar way for top level config items that are lists, str, int and float.

This will also happen if you miss anything from a dict within the config (eg if `config.forecast_div_mult_estimate` is a dict, then any keys present in this dict in the default YAML, but not in the config will be added). Finally it will work for nested dicts, eg if any keys are missing from `config.instrument_weight_estimate['correlation_estimate']` then they'll be filled in from the default file. If something is a dict, or a nested dict, in the config but not in the default (or vice versa) then values won't be replaced and bad things could happen. It's better to keep your config files, and the default file, with matching structures (for the items you want to change at least!). Again this is a good argument for adding new parameters, and retaining the original ones.

Note this means that the config before, and after, it goes into a system object will probably be different; the latter will be populated with defaults.

```python
from sysdata.config.configdata import Config
my_config=Config()
print(my_config) ## empty config
```

```
 Config with elements:
```

Now within a system:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system(config=my_config)

print(system.config) ## full of defaults.
print(my_config) ## same object
```

```
 Config with elements: average_absolute_forecast, base_currency, buffer_method, buffer_size, buffer_trade_to_edge, forecast_cap, forecast_correlation_estimate, forecast_div_mult_estimate, forecast_div_multiplier, forecast_scalar, forecast_scalar_estimate, forecast_weight_estimate, instrument_correlation_estimate, instrument_div_mult_estimate, instrument_div_multiplier, instrument_weight_estimate, notional_trading_capital, percentage_vol_target, use_SR_costs, use_forecast_scale_estimates, use_forecast_weight_estimates, use_instrument_weight_estimates, volatility_calculation
```

Note this isn't enough for a working trading system as trading rules aren't populated by the defaults:


```python
system.accounts.portfolio()
```

```
# deleted full error trace
Exception: A system config needs to include trading_rules, unless rules are passed when object created
```


### Viewing configuration parameters

Regardless of whether we create the dictionary using a YAML file or interactively, we'll end up with a dictionary. The keys in the top level dictionary will become attributes of the config. We can then use dictionary keys or list positions to access any nested data. For example using the simple config above:

```python
my_config.optionone
my_config.optiontwo['a']
my_config.optionthree[0]
```


### Modifying configuration parameters

It's equally straightforward to modify a config. For example using the simple config above:

```python
my_config.optionone=1.0
my_config.optiontwo['d']=5.0
my_config.optionthree.append(6.3)
```

You can also add new top level configuration items:

```python
my_config.optionfour=20.0
setattr(my_config, "optionfour", 20.0) ## if you prefer
```

Or remove them:

```python
del(my_config.optionone)
```

With real configs you need to be careful with nested parameters:


```python
config.instrument_div_multiplier=1.1 ## not nested, no problem

## Heres an example of how you'd change a nested parameter
## If the element doesn't yet exist in your config
## If the element did exist, then obviously doing this would overwrite all other parameters in the config - so don't do it!

config.volatility_calculation=dict(days=20)

## If it does exist you can do this instead:
config.volatility_calculation['days']=20
```

This is especially true if you're changing the config that has been included within a system, which will already include all the defaults:

```python
system.config.instrument_div_multiplier=1.1 ## not nested, no problem

## If we change anything that is nested, we need to change just one element to avoid clearing the defaults:
# So, do this:
system.config.volatility_calculation['days']=20

# Do NOT do this:
# system.config.volatility_calculation=dict(days=20)
```


### Using configuration in a system

Once we're happy with our configuration we can use it in a system:

```python
from systems.provided.futures_chapter15.basesystem import futures_system
system=futures_system(config=my_config)
```
Note it's only when a config is included i