core/data-tools/ — Data Models, API Tools & LLM Integration

Financial data fetching, caching, LLM provider abstraction, and utility functions

This sub-module bundles the utility layers that every agent depends on: Pydantic data models for financial data, a caching layer to avoid redundant API calls, a multi-provider LLM factory, CLI argument parsing with interactive selection, and display formatting utilities.

14
Providers
8
API Functions
5
Cache Types
6
Pydantic Models

What is this & why does it exist?

This sub-module bundles the utility layers that every agent depends on but that have no business logic of their own. tools/api.py wraps the Financial Datasets API with functions like get_prices(), get_financial_metrics(), and get_insider_trades(). data/cache.py provides in-memory caching to avoid redundant API calls within a single run. llm/models.py abstracts 14 LLM providers (OpenAI, Groq, Anthropic, DeepSeek, Google, Ollama, etc.) behind a single get_model() factory. cli/input.py handles argument parsing with interactive agent and model selection via questionary.

Real-world analogy

These utilities are the backstage crew of a theater production. The audience (user) sees the actors (agents) performing, but behind the scenes: stagehands (tools/api) fetch props (financial data), the wardrobe department (data/models) ensures every prop fits the right shape, the lighting crew (llm/models) controls the spotlights (LLM calls) across different brands of equipment, and the stage manager (cli) runs the show order. The cache is like a props table backstage — once a stagehand fetches a prop, it stays on the table so nobody has to run back to the warehouse.

Why Is It Designed This Way?

The key design decision is separation of infrastructure from intelligence. Agents should focus on analysis logic, not HTTP retries or provider-specific SDK quirks. The get_model() factory uses a simple if/elif chain rather than a plugin registry because the provider list is known at compile time and each provider has unique initialization logic (different env vars, base URLs, authentication flows). The cache uses in-memory dicts rather than Redis/disk because runs are short-lived (seconds to minutes) and the cache only needs to survive within a single run_hedge_fund() invocation. Pydantic models enforce data contracts at the API boundary so agents never receive malformed data.

Design ChoiceWhat Was ChosenWhat Was RejectedWhy
LLM abstractionFactory function with if/elifAbstract base class + pluginsEach provider has unique init logic; a factory is simpler and providers are enumerated at compile time
Caching strategyIn-memory dict singletonRedis / disk-backed cacheRuns last seconds-to-minutes; no persistence needed; zero infrastructure dependency
API data modelsPydantic BaseModelPlain dicts / dataclassesPydantic validates API responses at the boundary and provides model_dump() for serialization
Rate limitingLinear backoff (60s + 30s*n)Exponential backoffFinancial API has strict rate limits; linear backoff avoids thundering herd while being predictable

Contents

Architecture

The data-tools layer sits between agents and external services. Every agent calls utility functions; those functions check the cache, call APIs if needed, validate responses through Pydantic models, and return typed objects.

flowchart TD
  subgraph Agents
    A1["Analyst Agents"]
    RM["Risk Manager"]
    PM["Portfolio Manager"]
  end
  subgraph DataTools["Data and Tools Layer"]
    API["tools/api.py\n(get_prices, get_metrics, ...)"]
    CACHE["data/cache.py\n(in-memory singleton)"]
    MODELS["data/models.py\n(Pydantic validation)"]
    LLM["llm/models.py\n(get_model factory)"]
    CLI["cli/input.py\n(arg parsing)"]
  end
  subgraph External
    FIN["Financial Datasets API"]
    OPENAI["OpenAI / Groq / Anthropic\n/ 11 more providers"]
  end
  A1 --> API
  A1 --> LLM
  RM --> LLM
  PM --> LLM
  API --> CACHE
  CACHE -->|"miss"| FIN
  API --> MODELS
  LLM --> OPENAI
  CLI -->|"selected analysts\nmodel config"| A1

Data Flow

A typical API call follows: agent calls get_prices() → cache check (hit returns immediately) → HTTP request with retry → Pydantic parse → cache store → return typed list.

sequenceDiagram
  participant Agent
  participant API as tools/api.py
  participant Cache as data/cache.py
  participant Ext as Financial Datasets API
  participant Models as data/models.py

  Agent->>API: get_prices("AAPL", "2025-01-01", "2025-12-31")
  API->>Cache: get_prices(cache_key)
  alt Cache Hit
    Cache-->>API: cached list of dicts
    API-->>Agent: list of Price objects
  else Cache Miss
    Cache-->>API: None
    API->>Ext: GET /prices/?ticker=AAPL...
    Ext-->>API: JSON response
    API->>Models: PriceResponse(**json)
    Models-->>API: validated PriceResponse
    API->>Cache: set_prices(key, data)
    API-->>Agent: list of Price objects
  end

Key Classes & Functions

Class / FunctionFilePurpose
Pricedata/models.pyPydantic model for daily OHLCV price data (open, close, high, low, volume, time)
FinancialMetricsdata/models.pyPydantic model with 30+ financial ratio fields (P/E, P/B, margins, growth, etc.)
LineItemdata/models.pyPydantic model with extra="allow" for dynamic financial statement fields
Cachedata/cache.pyIn-memory cache with 5 typed stores (prices, metrics, line items, insider trades, news)
_merge_data()data/cache.pyDeduplicates by key field when appending new data to existing cache entries
get_cache()data/cache.pyReturns the global Cache singleton instance
get_prices()tools/api.pyFetches daily price data with cache-through and Pydantic validation
get_financial_metrics()tools/api.pyFetches financial ratios and metrics with configurable period and limit
_make_api_request()tools/api.pyHTTP wrapper with linear-backoff retry on 429 rate limits (up to 3 retries)
ModelProviderllm/models.pyEnum of 14 supported LLM providers (OpenAI, Anthropic, Groq, DeepSeek, etc.)
LLMModelllm/models.pyPydantic model for model configs with has_json_mode(), is_deepseek(), etc.
get_model()llm/models.pyFactory that returns a LangChain chat model for the given provider and model name
ANALYST_CONFIGutils/analysts.pySingle source of truth for all 19 analyst agents (names, styles, functions, ordering)
parse_cli_inputs()cli/input.pyParses CLI args and runs interactive questionary prompts for analyst/model selection

Data Structures

TypeNameDescription
PydanticPriceOHLCV bar: open, close, high, low, volume (int), time (str)
PydanticFinancialMetrics30+ nullable float fields covering valuation, profitability, efficiency, leverage, and growth
PydanticLineItemDynamic model (extra="allow") for arbitrary financial statement line items
EnumModelProvider14 members: OPENAI, ANTHROPIC, GROQ, DEEPSEEK, GOOGLE, OLLAMA, OPENROUTER, KIMI, XAI, GIGACHAT, AZURE_OPENAI, ALIBABA, META, MISTRAL
PydanticLLMModeldisplay_name, model_name, provider; helpers for JSON mode detection and provider checks
dictANALYST_CONFIGKey → {display_name, description, investing_style, agent_func, type, order}

Interactions

DirectionPartnerInterfaceDescription
←All analyst agentsget_prices(), get_financial_metrics()Agents call API functions to fetch market data for their analysis
←All agentsget_model()Every agent calls the factory to get a LangChain LLM instance matching user config
→Financial Datasets APIREST GET/POST_make_api_request() handles HTTP with retry logic for rate limits
→LLM Provider APIsLangChain SDKget_model() instantiates provider-specific LangChain classes
←main.pyparse_cli_inputs()CLI entry point uses input module for argument parsing and interactive selection
→Orchestration" href="../graph/index.html">graph/state.pyAgentState.metadataModel name and provider are stored in metadata and read by agents via get_model()
→utils/displayprint_trading_output()Formats final results with colorama styling for terminal output

Configuration

Knob / ParameterDefaultDescription
FINANCIAL_DATASETS_API_KEYenv varAPI key for financialdatasets.ai; set in .env
OPENAI_API_KEYenv varOpenAI API key; also supports OPENAI_API_BASE for custom endpoints
GROQ_API_KEYenv varGroq API key for fast inference
ANTHROPIC_API_KEYenv varAnthropic API key for Claude models
OLLAMA_HOSTlocalhostHostname for Ollama server (useful for Docker on macOS)
max_retries3Number of retry attempts on 429 rate limit responses

Directory Map

Source tree for data-tools
src/data/
  models.py              ← Pydantic models: Price, FinancialMetrics, LineItem, InsiderTrade, etc.
  cache.py               ← In-memory Cache singleton with 5 typed stores and dedup merge
src/tools/
  api.py                 ← Financial Datasets API wrappers with cache-through and retry
src/llm/
  models.py              ← ModelProvider enum, LLMModel config, get_model() factory
  api_models.json        ← JSON catalog of available cloud LLM models
  ollama_models.json     ← JSON catalog of local Ollama models
src/utils/
  analysts.py            ← ANALYST_CONFIG single source of truth for 19 analysts
  display.py             ← Terminal formatting with colorama
  progress.py            ← Progress spinner/bar during workflow execution
  visualize.py           ← Graph-to-PNG rendering utility
src/cli/
  input.py               ← Argument parsing + interactive questionary prompts

Annotated Code Walkthrough

The most important code in this sub-module, annotated with WHY each piece exists:

get_model() — LLM Provider Factory (llm/models.py)

def get_model(model_name: str, model_provider: ModelProvider,
               api_keys: dict = None):
    if model_provider == ModelProvider.GROQ:
        # WHY: Check api_keys dict first (webapp passes keys via API),
        # fall back to env var (CLI usage)
        api_key = (api_keys or {}).get("GROQ_API_KEY") or os.getenv("GROQ_API_KEY")
        if not api_key:
            raise ValueError("Groq API key not found.")
        return ChatGroq(model=model_name, api_key=api_key)

    elif model_provider == ModelProvider.OPENAI:
        api_key = (api_keys or {}).get("OPENAI_API_KEY") or os.getenv("OPENAI_API_KEY")
        base_url = os.getenv("OPENAI_API_BASE")  # WHY: Supports custom endpoints (Azure proxy, etc.)
        if not api_key:
            raise ValueError("OpenAI API key not found.")
        return ChatOpenAI(model=model_name, api_key=api_key, base_url=base_url)

    elif model_provider == ModelProvider.OLLAMA:
        # WHY: Ollama doesn't need an API key - it's local inference
        ollama_host = os.getenv("OLLAMA_HOST", "localhost")
        base_url = os.getenv("OLLAMA_BASE_URL", f"http://{ollama_host}:11434")
        return ChatOllama(model=model_name, base_url=base_url)

    elif model_provider == ModelProvider.OPENROUTER:
        api_key = (api_keys or {}).get("OPENROUTER_API_KEY") or os.getenv("OPENROUTER_API_KEY")
        # WHY: OpenRouter requires extra headers for attribution/ranking
        return ChatOpenAI(
            model=model_name, openai_api_key=api_key,
            openai_api_base="https://openrouter.ai/api/v1",
            model_kwargs={"extra_headers": {"HTTP-Referer": site_url, "X-Title": site_name}}
        )
    # ... 10 more providers follow the same pattern

Line-by-line:

  • Dual key lookup (api_keys or {}).get() or os.getenv(): The webapp passes API keys via the request body; the CLI reads from .env. This pattern supports both without branching at the call site.
  • Provider-specific classes: Each provider uses its own LangChain class (ChatGroq, ChatOpenAI, ChatAnthropic) because they have different constructors, auth flows, and parameters.
  • Ollama special case: No API key needed — it runs locally. The OLLAMA_HOST env var handles Docker-on-macOS where localhost doesn’t reach the container.
  • OpenRouter headers: OpenRouter aggregates models from many providers; the HTTP-Referer and X-Title headers are required for their attribution system.

Edge Cases & Boundaries

Edge CaseWhat HappensWhere in CodeRisk Level
API returns 429 (rate limited)Linear backoff retry: 60s, 90s, 120s; returns final response after max_retries_make_api_requestMedium
API returns invalid JSONPydantic validation fails; function logs warning and returns empty listget_prices:L85Low
Missing API key for providerValueError raised with descriptive message naming the expected env varget_modelHigh
Cache key collisionComposite keys include ticker + dates + params to prevent false hitsget_prices:L70Low
Unsupported model providerValueError raised listing all supported providersget_model:endMedium

Code Examples

Example: Using the LLM factory

from src.llm.models import get_model, ModelProvider

# Cloud provider (reads OPENAI_API_KEY from env)
llm = get_model("gpt-4.1", ModelProvider.OPENAI)
response = llm.invoke("Summarize AAPL Q4 earnings")

# Local Ollama
llm = get_model("llama3", ModelProvider.OLLAMA)

# With explicit API keys (webapp pattern)
llm = get_model("claude-sonnet-4-20250514", ModelProvider.ANTHROPIC,
                 api_keys={"ANTHROPIC_API_KEY": "sk-ant-..."})

Example: Fetching cached financial data

from src.tools.api import get_prices, get_financial_metrics

# First call hits the API and caches
prices = get_prices("AAPL", "2025-01-01", "2025-06-30")

# Second call with same params returns from cache instantly
prices_again = get_prices("AAPL", "2025-01-01", "2025-06-30")

# Different date range triggers a new API call
metrics = get_financial_metrics("AAPL", end_date="2025-06-30", period="ttm", limit=5)