backtesting — Historical Backtesting Engine

🎯 Why This Module Exists

Without backtesting, the system would be a black box — you’d have no way to evaluate whether the AI agents actually make profitable decisions or if their signals are noise. The backtesting module exists to provide empirical validation of the agent pipeline by replaying it against historical market data, day by day. Unlike simple “buy and check later” tests, this engine runs the full LangGraph agent workflow for each trading day, making decisions based only on data available at that point in time (no look-ahead bias). It tracks Portfolio Tracking & Performance Metrics" href="portfolio/index.html">portfolio positions (long and short), margin requirements, realized/unrealized P&L, and calculates industry-standard metrics (Sharpe ratio, maximum drawdown, cumulative returns). The module also compares performance against benchmarks (S&P 500). This isn’t just a testing tool — it’s how you discover which combinations of agents perform best in different market conditions.

🚶 Step-by-Step Narrated Walkthrough

Follow a single complete backtest run as it flows through the entire backtesting pipeline:

1. CLI Parsing — The user runs the CLI (cli.py). argparse collects tickers, date range, initial capital, and margin requirement. Interactive prompts via questionary let the user pick analysts and an LLM model (OpenAI, Anthropic, Ollama, etc.).
2. Engine Construction — A BacktestEngine is instantiated. It creates a fresh Portfolio with the given initial cash and margin requirement, plus a TradeExecutor, AgentController, PerformanceMetricsCalculator, OutputBuilder, and BenchmarkCalculator.
3. Data Prefetch — Before the day loop starts, _prefetch_data() calls the financial APIs to cache one year of price history, financial metrics, insider trades, and company news for every ticker—plus SPY for benchmarking. This avoids redundant API calls during the loop.
4. Day Loop Begins — pd.date_range(start, end, freq="B") generates every business day. For each day, a lookback_start is computed as one month prior, establishing the data window the agent will see (no future data).
5. Price Retrieval — For each ticker, the engine calls get_price_data() to fetch the closing price for the current day. If any ticker has missing data, the entire day is skipped—ensuring only complete market days are simulated.
6. Agent Invocation — AgentController.run_agent() converts the Portfolio object into a plain PortfolioSnapshot dict, then invokes the LangGraph agent pipeline. The agent sees only data from lookback_start to current_date. Returned decisions are normalized: actions coerced to the Action enum (buy/sell/short/cover/hold), quantities to floats, and missing tickers default to hold.
7. Trade Execution — For each ticker, TradeExecutor.execute_trade() dispatches to the appropriate Portfolio method. Long buys check available cash (partial fills if insufficient). Short opens check margin availability. Sells and covers are capped at current position size. Each method returns the actual executed quantity.
8. Valuation & Exposure — calculate_portfolio_value() computes total value as cash + long market value − short market value. compute_exposures() calculates long exposure, short exposure, gross/net exposure, and the long/short ratio for risk monitoring.
9. Output & Metrics — OutputBuilder.build_day_rows() formats each ticker’s action, position, and a summary row with total value, return percentage, Sharpe, Sortino, max drawdown, and benchmark return. Rows are prepended so the latest day appears first. After three days of data accumulate, PerformanceMetricsCalculator.compute_metrics() calculates annualized Sharpe (excess returns / std dev × √252), Sortino (excess / downside dev), and max drawdown (peak-to-trough percentage).
10. Final Report — After all days are processed, run_backtest() returns the final PerformanceMetrics dict. The CLI prints total return, Sharpe ratio, Sortino ratio, and max drawdown with the date it occurred. The portfolio value time series is also available via get_portfolio_values() for downstream charting.

Day-by-Day Simulation with Look-Ahead Prevention

The Problem: How do you evaluate an AI trading system without letting it “cheat” by seeing future data? A naive approach might feed the agent the entire historical dataset, but this produces unrealistically optimistic results.

The Approach: The engine iterates over business days using pd.date_range(freq="B"). For each day, it computes a lookback_start one month prior. The agent only receives data within [lookback_start, current_date]. Price data is fetched for [previous_date, current_date] to get the closing price. The portfolio snapshot passed to the agent reflects only the state before today’s trades execute. This three-layer isolation—date windowing, snapshot-before-trade, and sequential day processing—ensures no future information leaks into decisions.

Why This Algorithm: Vectorized backtesting (apply signals to a price matrix at once) is much faster but cannot capture the feedback loop where today’s portfolio state affects tomorrow’s decisions. Since the agent pipeline is stateful (it considers current positions, cash, and margin), day-by-day simulation is the only correct approach. The performance cost is dominated by LLM API calls, not the loop itself.

flowchart TD
    Start["Start: Generate business day range"] --> Loop{"Next business day?"}
    Loop -->|"Yes"| Window["Compute lookback window\ncurrent_date - 1 month"]
    Window --> Prices["Fetch closing prices\nfor current_date"]
    Prices --> Check{"All tickers\nhave data?"}
    Check -->|"No"| Skip["Skip day\nno partial data"] --> Loop
    Check -->|"Yes"| Snapshot["Create portfolio snapshot\nstate BEFORE today"]
    Snapshot --> Agent["Run agent with\nlookback window only"]
    Agent --> Normalize["Normalize decisions\ncoerce to Action enum"]
    Normalize --> Execute["Execute trades\nupdate portfolio"]
    Execute --> Value["Calculate portfolio value\nand exposures"]
    Value --> Metrics["Update performance\nmetrics if 3+ days"]
    Metrics --> Output["Print daily report\nlatest day first"]
    Output --> Loop
    Loop -->|"No more days"| Done["Return final\nPerformanceMetrics"]

Hot Paths

The most performance-sensitive code paths in this module:

• LLM Agent Invocation — ~1–30 seconds per day depending on model and number of analysts. This dominates total runtime by orders of magnitude. A 6-month backtest with 5 tickers and 8 analysts can take hours. The engine itself adds negligible overhead.
• Data Prefetch — O(T) where T = number of tickers. Each ticker triggers 4 API calls (prices, financials, insider trades, news). With caching, subsequent days reuse prefetched data. SPY is also prefetched for benchmarking.
• Metrics Computation — O(D) where D = number of days elapsed. Sharpe/Sortino/drawdown iterate over the full portfolio value series each day. For typical backtests (60–250 days), this completes in microseconds.

Resource Usage

• Memory: Portfolio state is O(T) — one PositionState dict per ticker. The portfolio_values list grows by one point per day (O(D)). Table rows accumulate for display (O(T × D)). No data is freed during the run; total memory is modest for typical backtests.
• I/O: Financial data APIs are the primary I/O. Prefetching warms the internal cache. After prefetch, the day loop reads only from cache for market data. LLM calls are the dominant I/O cost, both in latency and in token spend.