Using Interactive Brokers Data for Backtests: Coverage and Data Limits

# backtesting.ibkr


Interactive Brokers (REST) Backtesting
======================================

.. meta::
   :description: LumiBot supports backtesting with Interactive Brokers data providers.

LumiBot supports backtesting with **Interactive Brokers data providers**.

The primary data path uses Client Portal (REST) via the LumiBot Data Downloader.

Note: second-level and tick-level history are not yet a fully-supported end-to-end workflow in the open-source
distribution. (Some internal deployments may have additional adapters behind the same external ``/ibkr/*`` contract.)

For **expired futures** contract discovery (conids), LumiBot checks its mirrored contract registry first.
When REST does not list the requested expired month, a downloader with the read-only
``/ibkr/tws/secdef/contracts`` endpoint can resolve it through its existing TWS gateway.
The root, venue, currency and expiration month must match one unambiguous futures contract.
Discovery cannot extend IBKR's historical-data retention limits.

NG, CL and MCL discovery uses the exact last-trade date because it precedes
the delivery month. A completed TWS lookup without a matching identity has a
15-minute retry cooldown; a newly recovered positive registry entry takes
precedence. Timeouts and disconnects do not establish identity absence, and
unresolved identities never establish that historical prices did not exist.

Missing continuous-futures roll contracts are reported as partial history, even when later contracts supply bars.
Check the actual number of completed bars at the simulated decision time before applying a lookback indicator.
Shared registry writes preserve unrelated current identities and retry concurrent publication conflicts.
Futures weekend closure runs from Friday 17:00 through Sunday 18:00 New York time, across daylight-saving changes.

Downloader queue deadlines cover submission, polling, retry backoff and local concurrency waits together.
Normal requests allow three configured timeout windows. Requests with a finite attempt cap use that many windows;
resubmission does not restart the total deadline. If a replacement downloader explicitly reports
that an accepted read request no longer exists, the client resubmits the same logical read
after a bounded pause. Transient status errors keep waiting on the existing request. Already downloaded valid bars remain available for a later retry.

Status
------

IBKR REST backtesting is under active development and is not yet a fully-supported public workflow in the open-source
distribution.

If you want early access or have a specific use case, please open an issue (or contact the maintainers) so we can
prioritize it.

Quick Start
-----------

Select IBKR as the backtesting data source:

.. code-block:: bash

   export BACKTESTING_DATA_SOURCE="ibkr"

Supported Data
--------------

- **Futures**: US futures across CME/CBOT/COMEX/NYMEX via IBKR historical endpoints (minute/hour/day bars).
- **Spot crypto**: IBKR crypto bars (availability depends on region and IBKR product support).
- **Stocks / Indexes (day bars)**: supported in routed backtests (for example mixed Theta+IBKR routing).

Stock, index and futures intraday historical requests stop 20 minutes before
the backtest's captured start-of-run clock. Explicit and continuous futures use the same stable boundary,
including the pager's overlap offset. A window wholly inside that unavailable
interval returns empty bars with partial history health; unavailable prices
are never cached as confirmed absence. Daily bars, older historical windows
and real-time snapshot APIs retain their separate behavior.

Portfolio Valuation (Stocks/Indexes)
------------------------------------

Daily-cadence backtests (for example ``sleeptime = "1D"``) value stock and index positions on the daily
series. Intraday backtests value them on the finest intraday bars loaded for that asset (the bars the
strategy requested and its fills use): the bar that has just completed marks at its close, a bar still forming
marks at its open. If the strategy's minute history ends before the current time, a small window of minute bars
is fetched first, the same request the strategy itself makes. Strategies that only use daily bars never fetch
minute history for valuation.

Daily futures candles aggregate intraday bars whose start time is inside the session,
including the opening boundary and excluding the closing boundary. A bar starting at
the close belongs to the next session and cannot change the completed candle's prices
or volume. This applies to both hourly aggregation and the minute fallback.

During known daily maintenance and weekend closures, futures valuation can
retain the last completed trade close. This fallback cannot bridge a missing
open-market interval or supply an executable bar.

Daily Stocks/Indexes: Warmup + Corporate Actions
------------------------------------------------

TWS daily bars use calendar-date labels encoded at UTC midnight. LumiBot preserves that session date
before converting to New York and exposes the candle at the existing daily close. New stock/index
daily cache filenames use ``_SESSION_DATE_V2.parquet`` to rebuild mixed legacy daily files from real
provider data. Old files remain available to older package versions; intraday and futures cache keys
are unchanged. Provider gaps still report partial history and are never filled with invented prices.


For routed daily stock/index backtests, LumiBot prefetches the full computed lookback window so long
lookbacks (for example 200-day SMA signals) are not under-warmed near the backtest start.

IBKR day-bar payloads do not include corporate-action columns directly. LumiBot enriches cached IBKR
daily equity bars with ``dividend`` and ``stock_splits`` values using Yahoo actions as a best-effort
source so split/dividend accounting remains available in backtests.

How History Is Downloaded (stocks and indexes)
----------------------------------------------

IBKR returns at most about 1,000 bars per request, so LumiBot walks backwards page by page.

- **Weekends, holidays and nights.** A 1-minute page covers 1,000 minutes (16.7 hours). A page that falls
  entirely inside closed-market time comes back empty; LumiBot steps over it and keeps going instead of treating it
  as the start of history. US indexes such as SPX only print 09:30 to 16:00 ET, so the same applies every night.
- **New listings.** When a daily page reaches back before the first bar IBKR holds, IBKR answers
  ``Chart data unavailable``. LumiBot retries with a smaller page and keeps the real bars it already has, so a
  fund listed last year still gets its full daily history.
- **A failed older page** keeps the newer real bars already downloaded; the missing older part is not faked and is
  retried by a later run.
- **Delayed feed.** IBKR stock, index and futures history can lag real time, so intraday requests
  stop 20 minutes before the captured start-of-run clock. A strategy decision requiring newer
  unavailable prices makes the run incomplete; it does not silently qualify a shortened result.
- **Daily windows** up to 993 days are one request sized to the window; longer windows use 5-year pages.
- **Dividends.** IBKR history has no corporate actions, so LumiBot adds dividends and splits to IBKR daily stock bars
  from a free corporate-actions source. BotSpot Auto backtests credit a held stock's dividend on its ex-date from
  those daily bars.
- **Holes in cached minute bars.** When the cache has bars on both sides of a missing session (for example from two
  earlier backtests, or a download that was stopped), LumiBot downloads each missing session instead of skipping it.
  A session with no trades at all is remembered for a day so it is not requested again by every backtest.
  Gap checks handle nanosecond, microsecond, millisecond and second cache timestamps consistently;
  existing Parquet caches do not need to be deleted or rewritten.

Continuous Futures: History and Held Contracts
---------------------------------------------

A continuous series supplies signal history. In IBKR backtests, an order submitted
for that series binds to the selected physical expiry. The position and its protective
orders retain that expiry after the continuous chart moves to the next contract.
There is no automatic roll trade and no profit from merely switching chart series.

To roll a position, close its ``position.asset`` and open the next contract explicitly.
A root-symbol closing order targets the single matching held contract. Multiple matching
expiries, or a reversal spanning different expiries, require explicit contracts.
Normal fills, fees and price availability apply to both legs.

Futures Exchange Routing (auto + override)
------------------------------------------

For futures and continuous futures (``asset_type="future"`` / ``asset_type="cont_future"``), LumiBot supports an
optional ``exchange=...`` parameter on the Strategy data methods:

- ``get_historical_prices(..., exchange=...)``
- ``get_last_price(..., exchange=...)``
- ``get_quote(..., exchange=...)``

When ``exchange`` is omitted, LumiBot attempts to resolve the correct futures exchange automatically via IBKR secdef
search (preferring USD + US venues). If results are ambiguous, you must pass ``exchange=...`` explicitly.

Expired Futures Contracts (conids)
----------------------------------

IBKR futures history requires a contract identifier (``conid``). IBKR Client Portal cannot reliably discover ``conid``
values for **expired** futures contracts, which makes explicit-contract backtests fail unless the mapping is already
known.

LumiBot supports an offline conid registry (cache-backed, optionally S3-mirrored):

- ``LUMIBOT_CACHE_FOLDER/ibkr/conids.json``

The registry is expected to **grow automatically over time** for new contracts as backtests run (using IBKR REST
endpoints). Very old expired contracts may still require a one-time offline backfill in internal deployments.

Internal runbook (engineering): ``docs/investigations/2026-01-18_IBKR_EXPIRED_FUTURES_CONID_BACKFILL.md``.

Caching
-------

IBKR backtests cache historical bars as Parquet:


- Optional S3 mirroring: configured via the standard ``LUMIBOT_CACHE_*`` variables (see :ref:`environment_variables`).

For US stock and index bars, LumiBot checks for cache gaps after loading the
series. Daily bars are compared with completed NYSE sessions and exact missing
session groups are repaired in small bounded windows under a 45-second
per-series deadline. Hourly bars are checked for internal holes longer than
seven days and repaired in bounded 2000-hour pages under a five-minute
per-series deadline. These checks never turn partial data into a backtest
exception, and complete warm caches do not call the downloader.

History health is classified as ``complete``, ``partial``,
``confirmed_no_data``, or ``transient_failure``. Only confirmed absence may
create a durable no-data marker. That marker includes a reason and retry
timestamp. Partial and transient responses receive an in-process cooldown so
one backtest does not repeat the same downloader request, while a later process
remains able to retry. Legacy, ambiguous, and expired markers are eligible for
lazy repair. When a real bar and a no-data marker share a timestamp, the real
bar always wins.

Backtest ``settings.json`` artifacts include a credential-free ``data_health``
summary with up to 100 missing-session dates, the full missing-session count,
and repair outcomes. Free-form provider errors remain in logs.

The summary distinguishes optional prefetch gaps from history actually required
by a strategy. ``required_complete=false`` means a decision lacked its requested
completed bars, a consumed contract segment was unresolved, a held position had
no valuation price, or a decision required an unavailable delayed-feed tail.
Consumers must not present such a run as a valid completed result. A fully
supplied strategy that legitimately chooses no trades remains valid.

Conid lookups also maintain cache files under ``LUMIBOT_CACHE_FOLDER/ibkr``:

- ``conids.json`` stores successful conid resolutions.
- ``conids_negative.json`` stores short-lived negative markers for symbols IBKR cannot resolve.

Negative conid markers prevent long backtests from repeatedly retrying permanently unavailable symbols, such as
defunct equities returned by an external universe API. These lookup failures are treated as terminal no-data
conditions for the affected request window, not as synthetic price data.

Multi-provider routing (Theta + IBKR)
-------------------------------------

To use multiple providers in a single backtest (example: ThetaData for options/stocks/indexes and IBKR for futures/crypto), set a JSON mapping in ``BACKTESTING_DATA_SOURCE``:

.. code-block:: bash

   export BACKTESTING_DATA_SOURCE='{"default":"thetadata","stock":"thetadata","option":"thetadata","index":"thetadata","future":"ibkr","cont_future":"ibkr","crypto":"ibkr","crypto_future":"ibkr"}'

Routing values are case/whitespace/_/- insensitive. For crypto, you may also route to documented CCXT backtesting paths by using either ``"ccxt"`` (auto-select exchange) or a supported CCXT backtesting exchange id directly (for example: ``"kraken"`` or ``"binance"``).

Crypto futures and perpetuals
-----------------------------

For ``Asset.AssetType.CRYPTO_FUTURE`` backtests, LumiBot can load spot crypto history as a price proxy while preserving the strategy-facing futures asset for orders and positions. USDT contracts such as ``BTCUSDT``, ``ETHUSDT``, and ``SOLUSDT`` resolve to the corresponding USD spot proxy (``BTC/USD``, ``ETH/USD``, ``SOL/USD``) because common crypto perpetual venues keep the futures price close to spot. The log will state when a USD spot proxy is used.

Market Data Subscriptions (IBKR)
--------------------------------

IBKR requires appropriate **market data entitlements** to access market data via the API. IBKR notes that historical bars are part of Level 1 entitlements and that **crypto does not require additional market data subscriptions**:

- https://www.interactivebrokers.com/campus/ibkr-api-page/market-data-subscriptions/

For **CME futures** (ES/MES/NQ/MNQ), note that the cheap **CME S&P Indices** subscription is for index data, not futures contracts. Professional subscriber pricing and package availability can differ, so confirm your exact costs in the IBKR Market Data Subscriptions page and pricing table:

- https://www.interactivebrokers.com/en/pricing/market-data-pricing.php

Authentication / Session Behavior
---------------------------------

The Client Portal Gateway is session-based; if the session becomes unauthenticated, the gateway must be re-authenticated. IBKR documents the expected authentication lifecycle and recommends using ``/iserver/auth/ssodh/init`` to re-authenticate in most scenarios:

- https://www.interactivebrokers.com/campus/trading-lessons/launching-and-authenticating-the-gateway/

Configuration Notes
-------------------

Common environment variables for IBKR REST backtesting:

- ``IBKR_HISTORY_SOURCE`` (default: ``Trades``)
- ``IBKR_FUTURES_EXCHANGE`` (default: ``CME``; fallback when auto-routing fails)
- ``IBKR_CRYPTO_VENUE`` (default: ``ZEROHASH``)
- ``LUMIBOT_IBKR_ENABLE_FUTURES_BID_ASK`` (default: disabled; opt-in quote derivation for futures)

See :ref:`environment_variables` for details.


Continuous history and tradable positions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Continuous-futures history switches contracts on the configured roll schedule.
An IBKR backtest order resolves to a physical expiry when submitted. Its fills,
protective children and held position keep that expiry; a change in the history
series does not itself trade or change position value. Close the held contract
and submit the replacement explicitly when the strategy intends to roll. Both
fills use their own observed contract prices and configured fees.

A root-symbol closing order targets a single matching held expiry. If several
opposite positions exist, or a reversal would span different expiries, use
explicit ``position.asset`` contracts and separate orders. The simulator rejects
ambiguous operations rather than silently choosing a different financial result.