core/data-tools/ — Data Models, API Tools & LLM Integration
Financial data fetching, caching, LLM provider abstraction, and utility functions
This sub-module bundles the utility layers that every agent depends on: Pydantic data models for financial data, a caching layer to avoid redundant API calls, a multi-provider LLM factory, CLI argument parsing with interactive selection, and display formatting utilities.
What is this & why does it exist?
This sub-module bundles the utility layers that every agent depends on but that have no business logic of their own. tools/api.py wraps the Financial Datasets API with functions like get_prices(), get_financial_metrics(), and get_insider_trades(). data/cache.py provides in-memory caching to avoid redundant API calls within a single run. llm/models.py abstracts 14 LLM providers (OpenAI, Groq, Anthropic, DeepSeek, Google, Ollama, etc.) behind a single get_model() factory. cli/input.py handles argument parsing with interactive agent and model selection via questionary.
Real-world analogy
These utilities are the backstage crew of a theater production. The audience (user) sees the actors (agents) performing, but behind the scenes: stagehands (tools/api) fetch props (financial data), the wardrobe department (data/models) ensures every prop fits the right shape, the lighting crew (llm/models) controls the spotlights (LLM calls) across different brands of equipment, and the stage manager (cli) runs the show order. The cache is like a props table backstage — once a stagehand fetches a prop, it stays on the table so nobody has to run back to the warehouse.
Why Is It Designed This Way?
The key design decision is separation of infrastructure from intelligence. Agents should focus on analysis logic, not HTTP retries or provider-specific SDK quirks. The get_model() factory uses a simple if/elif chain rather than a plugin registry because the provider list is known at compile time and each provider has unique initialization logic (different env vars, base URLs, authentication flows). The cache uses in-memory dicts rather than Redis/disk because runs are short-lived (seconds to minutes) and the cache only needs to survive within a single run_hedge_fund() invocation. Pydantic models enforce data contracts at the API boundary so agents never receive malformed data.
| Design Choice | What Was Chosen | What Was Rejected | Why |
|---|---|---|---|
| LLM abstraction | Factory function with if/elif | Abstract base class + plugins | Each provider has unique init logic; a factory is simpler and providers are enumerated at compile time |
| Caching strategy | In-memory dict singleton | Redis / disk-backed cache | Runs last seconds-to-minutes; no persistence needed; zero infrastructure dependency |
| API data models | Pydantic BaseModel | Plain dicts / dataclasses | Pydantic validates API responses at the boundary and provides model_dump() for serialization |
| Rate limiting | Linear backoff (60s + 30s*n) | Exponential backoff | Financial API has strict rate limits; linear backoff avoids thundering herd while being predictable |
Contents
Architecture
The data-tools layer sits between agents and external services. Every agent calls utility functions; those functions check the cache, call APIs if needed, validate responses through Pydantic models, and return typed objects.
flowchart TD
subgraph Agents
A1["Analyst Agents"]
RM["Risk Manager"]
PM["Portfolio Manager"]
end
subgraph DataTools["Data and Tools Layer"]
API["tools/api.py\n(get_prices, get_metrics, ...)"]
CACHE["data/cache.py\n(in-memory singleton)"]
MODELS["data/models.py\n(Pydantic validation)"]
LLM["llm/models.py\n(get_model factory)"]
CLI["cli/input.py\n(arg parsing)"]
end
subgraph External
FIN["Financial Datasets API"]
OPENAI["OpenAI / Groq / Anthropic\n/ 11 more providers"]
end
A1 --> API
A1 --> LLM
RM --> LLM
PM --> LLM
API --> CACHE
CACHE -->|"miss"| FIN
API --> MODELS
LLM --> OPENAI
CLI -->|"selected analysts\nmodel config"| A1
Data Flow
A typical API call follows: agent calls get_prices() → cache check (hit returns immediately) → HTTP request with retry → Pydantic parse → cache store → return typed list.
sequenceDiagram
participant Agent
participant API as tools/api.py
participant Cache as data/cache.py
participant Ext as Financial Datasets API
participant Models as data/models.py
Agent->>API: get_prices("AAPL", "2025-01-01", "2025-12-31")
API->>Cache: get_prices(cache_key)
alt Cache Hit
Cache-->>API: cached list of dicts
API-->>Agent: list of Price objects
else Cache Miss
Cache-->>API: None
API->>Ext: GET /prices/?ticker=AAPL...
Ext-->>API: JSON response
API->>Models: PriceResponse(**json)
Models-->>API: validated PriceResponse
API->>Cache: set_prices(key, data)
API-->>Agent: list of Price objects
end
Key Classes & Functions
| Class / Function | File | Purpose |
|---|---|---|
Price | data/models.py | Pydantic model for daily OHLCV price data (open, close, high, low, volume, time) |
FinancialMetrics | data/models.py | Pydantic model with 30+ financial ratio fields (P/E, P/B, margins, growth, etc.) |
LineItem | data/models.py | Pydantic model with extra="allow" for dynamic financial statement fields |
Cache | data/cache.py | In-memory cache with 5 typed stores (prices, metrics, line items, insider trades, news) |
_merge_data() | data/cache.py | Deduplicates by key field when appending new data to existing cache entries |
get_cache() | data/cache.py | Returns the global Cache singleton instance |
get_prices() | tools/api.py | Fetches daily price data with cache-through and Pydantic validation |
get_financial_metrics() | tools/api.py | Fetches financial ratios and metrics with configurable period and limit |
_make_api_request() | tools/api.py | HTTP wrapper with linear-backoff retry on 429 rate limits (up to 3 retries) |
ModelProvider | llm/models.py | Enum of 14 supported LLM providers (OpenAI, Anthropic, Groq, DeepSeek, etc.) |
LLMModel | llm/models.py | Pydantic model for model configs with has_json_mode(), is_deepseek(), etc. |
get_model() | llm/models.py | Factory that returns a LangChain chat model for the given provider and model name |
ANALYST_CONFIG | utils/analysts.py | Single source of truth for all 19 analyst agents (names, styles, functions, ordering) |
parse_cli_inputs() | cli/input.py | Parses CLI args and runs interactive questionary prompts for analyst/model selection |
Data Structures
| Type | Name | Description |
|---|---|---|
| Pydantic | Price | OHLCV bar: open, close, high, low, volume (int), time (str) |
| Pydantic | FinancialMetrics | 30+ nullable float fields covering valuation, profitability, efficiency, leverage, and growth |
| Pydantic | LineItem | Dynamic model (extra="allow") for arbitrary financial statement line items |
| Enum | ModelProvider | 14 members: OPENAI, ANTHROPIC, GROQ, DEEPSEEK, GOOGLE, OLLAMA, OPENROUTER, KIMI, XAI, GIGACHAT, AZURE_OPENAI, ALIBABA, META, MISTRAL |
| Pydantic | LLMModel | display_name, model_name, provider; helpers for JSON mode detection and provider checks |
| dict | ANALYST_CONFIG | Key → {display_name, description, investing_style, agent_func, type, order} |
Interactions
| Direction | Partner | Interface | Description |
|---|---|---|---|
| ← | All analyst agents | get_prices(), get_financial_metrics() | Agents call API functions to fetch market data for their analysis |
| ← | All agents | get_model() | Every agent calls the factory to get a LangChain LLM instance matching user config |
| → | Financial Datasets API | REST GET/POST | _make_api_request() handles HTTP with retry logic for rate limits |
| → | LLM Provider APIs | LangChain SDK | get_model() instantiates provider-specific LangChain classes |
| ← | main.py | parse_cli_inputs() | CLI entry point uses input module for argument parsing and interactive selection |
| → | Orchestration" href="../graph/index.html">graph/state.py | AgentState.metadata | Model name and provider are stored in metadata and read by agents via get_model() |
| → | utils/display | print_trading_output() | Formats final results with colorama styling for terminal output |
Configuration
| Knob / Parameter | Default | Description |
|---|---|---|
FINANCIAL_DATASETS_API_KEY | env var | API key for financialdatasets.ai; set in .env |
OPENAI_API_KEY | env var | OpenAI API key; also supports OPENAI_API_BASE for custom endpoints |
GROQ_API_KEY | env var | Groq API key for fast inference |
ANTHROPIC_API_KEY | env var | Anthropic API key for Claude models |
OLLAMA_HOST | localhost | Hostname for Ollama server (useful for Docker on macOS) |
max_retries | 3 | Number of retry attempts on 429 rate limit responses |
Directory Map
Source tree for data-tools
src/data/
models.py ← Pydantic models: Price, FinancialMetrics, LineItem, InsiderTrade, etc.
cache.py ← In-memory Cache singleton with 5 typed stores and dedup merge
src/tools/
api.py ← Financial Datasets API wrappers with cache-through and retry
src/llm/
models.py ← ModelProvider enum, LLMModel config, get_model() factory
api_models.json ← JSON catalog of available cloud LLM models
ollama_models.json ← JSON catalog of local Ollama models
src/utils/
analysts.py ← ANALYST_CONFIG single source of truth for 19 analysts
display.py ← Terminal formatting with colorama
progress.py ← Progress spinner/bar during workflow execution
visualize.py ← Graph-to-PNG rendering utility
src/cli/
input.py ← Argument parsing + interactive questionary prompts
Annotated Code Walkthrough
The most important code in this sub-module, annotated with WHY each piece exists:
get_model() — LLM Provider Factory (llm/models.py)
def get_model(model_name: str, model_provider: ModelProvider,
api_keys: dict = None):
if model_provider == ModelProvider.GROQ:
# WHY: Check api_keys dict first (webapp passes keys via API),
# fall back to env var (CLI usage)
api_key = (api_keys or {}).get("GROQ_API_KEY") or os.getenv("GROQ_API_KEY")
if not api_key:
raise ValueError("Groq API key not found.")
return ChatGroq(model=model_name, api_key=api_key)
elif model_provider == ModelProvider.OPENAI:
api_key = (api_keys or {}).get("OPENAI_API_KEY") or os.getenv("OPENAI_API_KEY")
base_url = os.getenv("OPENAI_API_BASE") # WHY: Supports custom endpoints (Azure proxy, etc.)
if not api_key:
raise ValueError("OpenAI API key not found.")
return ChatOpenAI(model=model_name, api_key=api_key, base_url=base_url)
elif model_provider == ModelProvider.OLLAMA:
# WHY: Ollama doesn't need an API key - it's local inference
ollama_host = os.getenv("OLLAMA_HOST", "localhost")
base_url = os.getenv("OLLAMA_BASE_URL", f"http://{ollama_host}:11434")
return ChatOllama(model=model_name, base_url=base_url)
elif model_provider == ModelProvider.OPENROUTER:
api_key = (api_keys or {}).get("OPENROUTER_API_KEY") or os.getenv("OPENROUTER_API_KEY")
# WHY: OpenRouter requires extra headers for attribution/ranking
return ChatOpenAI(
model=model_name, openai_api_key=api_key,
openai_api_base="https://openrouter.ai/api/v1",
model_kwargs={"extra_headers": {"HTTP-Referer": site_url, "X-Title": site_name}}
)
# ... 10 more providers follow the same pattern
Line-by-line:
- Dual key lookup
(api_keys or {}).get() or os.getenv(): The webapp passes API keys via the request body; the CLI reads from.env. This pattern supports both without branching at the call site. - Provider-specific classes: Each provider uses its own LangChain class (
ChatGroq,ChatOpenAI,ChatAnthropic) because they have different constructors, auth flows, and parameters. - Ollama special case: No API key needed — it runs locally. The
OLLAMA_HOSTenv var handles Docker-on-macOS where localhost doesn’t reach the container. - OpenRouter headers: OpenRouter aggregates models from many providers; the
HTTP-RefererandX-Titleheaders are required for their attribution system.
Edge Cases & Boundaries
| Edge Case | What Happens | Where in Code | Risk Level |
|---|---|---|---|
| API returns 429 (rate limited) | Linear backoff retry: 60s, 90s, 120s; returns final response after max_retries | _make_api_request | Medium |
| API returns invalid JSON | Pydantic validation fails; function logs warning and returns empty list | get_prices:L85 | Low |
| Missing API key for provider | ValueError raised with descriptive message naming the expected env var | get_model | High |
| Cache key collision | Composite keys include ticker + dates + params to prevent false hits | get_prices:L70 | Low |
| Unsupported model provider | ValueError raised listing all supported providers | get_model:end | Medium |
Code Examples
Example: Using the LLM factory
from src.llm.models import get_model, ModelProvider
# Cloud provider (reads OPENAI_API_KEY from env)
llm = get_model("gpt-4.1", ModelProvider.OPENAI)
response = llm.invoke("Summarize AAPL Q4 earnings")
# Local Ollama
llm = get_model("llama3", ModelProvider.OLLAMA)
# With explicit API keys (webapp pattern)
llm = get_model("claude-sonnet-4-20250514", ModelProvider.ANTHROPIC,
api_keys={"ANTHROPIC_API_KEY": "sk-ant-..."})
Example: Fetching cached financial data
from src.tools.api import get_prices, get_financial_metrics
# First call hits the API and caches
prices = get_prices("AAPL", "2025-01-01", "2025-06-30")
# Second call with same params returns from cache instantly
prices_again = get_prices("AAPL", "2025-01-01", "2025-06-30")
# Different date range triggers a new API call
metrics = get_financial_metrics("AAPL", end_date="2025-06-30", period="ttm", limit=5)