llms — Legacy completion-style LLM wrappers (text-in, text-out)

Legacy completion-style LLM wrappers (text-in, text-out)

Module langchain_classic/llms/ — Legacy completion-style LLM wrappers (text-in, text-out). Part of the Models I/O group.

🎯 Why This Module Exists

Wraps completion-style LLM providers (text-in, text-out) under a single BaseLLM abstraction. Each subclass owns provider-specific HTTP, auth, and parameter mapping. The base class owns retries, batching, caching (BaseCache), streaming, and emits standardised on_llm_start/on_llm_end/on_llm_new_token callbacks. Without this layer, every chain or agent would bake in provider-specific logic.

Source files

Sub-directories

Models I/O

Architecture

flowchart LR
  subgraph Core["langchain_classic/llms/"]
    BASE["Base abstraction"]
    IMPLS["Concrete implementations"]
    BASE --> IMPLS
  end
  subgraph Up["Upstream callers"]
    U1["chains/"]
    U2["agents/"]
  end
  subgraph Down["Downstream"]
    D1["langchain_core"]
    D2["External providers"]
  end
  Up --> Core
  Core --> Down

Data Flow

sequenceDiagram
  participant Caller as Caller
  participant Llms as llms
  participant Core as langchain_core
  participant Ext as External
  Caller->>Llms: invoke(input)
  Note over Llms: Validate & route
  Llms->>Core: delegate primitive
  Core->>Ext: provider call
  Ext-->>Core: response
  Core-->>Llms: typed output
  Llms-->>Caller: result

Design Decisions

Decision	Choice Made	Alternatives	Why This Approach
BaseLLM vs BaseChatModel	Two abstract bases	One base with mode flag	Completion and chat models have different message shapes; subclassing keeps each surface clean
Retries	tenacity-based; configurable per-call	Hand-rolled exponential backoff	tenacity is the standard library for this in Python
Caching	BaseCache pluggable (in-memory, SQLite, Redis, Cassandra)	Single in-memory dict	Production deployments need persistent cache
Streaming	Yield Generation chunks via callbacks	Return AsyncIterator only	Sync iteration broadly required by 2022-era integrations
Token counting	Optional; provider-specific	Always count via tiktoken	Not all models are OpenAI-tokenized; cost is too high to mandate

Concrete subclasses cover OpenAI, Anthropic, HuggingFace, AI21, Cohere, Vertex, Bedrock, Replicate, Together, Fireworks, ... plus self-hosted (Ollama, vLLM, LlamaCpp, GPT4All).

Step-by-Step Walkthrough

Follow a typical operation through the llms/ module:

Caller invokes the public surface — usually .invoke(input) on a Runnable, or a legacy __call__(inputs_dict) on a Chain subclass.
Pydantic validation — every entry point is a BaseModel; constructor args, input dicts, and tool args_schema are validated before any expensive work runs. Fail-fast on type errors.
CallbackManager.configure — a fresh manager is built per call, fans on_*_start events to all subscribers (LangSmith tracer, stdout handler, custom logging), and stamps a UUID4 run_id for tree reconstruction.
llms-specific logic — the module's core method runs: prompt rendering, retrieval, tool execution, memory load, document split, etc. See the deep-dive sections above.
Provider call — typically a single HTTP request to OpenAI/Anthropic/etc., or a local SDK call. Retries are handled by tenacity with exponential backoff and a configurable max-attempts knob.
Streaming fan-out — if the provider streams tokens, each chunk fires on_llm_new_token through the callback manager so downstream observers (UI, log, tracer) see partial output.
OutputParser pass — if a parser is attached, the raw provider output is parsed into a structured Python object; OutputParserException may trigger a retry chain (RetryWithErrorOutputParser).
Memory write-back — for chains/agents that have a BaseMemory attached, save_context(inputs, outputs) persists the turn before returning to the caller.
on_*_end fan-out — the callback manager closes the run span; tracers serialise the Run tree and ship it to LangSmith asynchronously.
Result returned — caller receives a typed object (AIMessage, dict, BaseModel, list[Document], ...) matching the module's declared output schema.

What Could Go Wrong

Failure Mode	Trigger	Detection	Recovery
Provider 5xx / rate-limit	OpenAI/Anthropic/etc. transient error	HTTP status from SDK	tenacity retries with exponential backoff up to `max_retries`; then surface to caller
Output parser failure	LLM returned malformed JSON / wrong schema	`OutputParserException` in parser	`RetryWithErrorOutputParser` re-prompts with the parser error included; or surface
Pydantic validation	Caller passes wrong-shaped dict	`ValidationError` at `__init__` / `invoke`	Surface immediately — no fallback is correct here
Callback handler exception	User-supplied tracer raises	Try/except around each fan-out	Logged and swallowed by default; `raise_error=True` to opt in to propagation
Provider auth missing	`OPENAI_API_KEY` unset	SDK `AuthenticationError`	Surface with a helpful message; no retry (would just fail again)

Performance Profile

Hot Paths

Provider HTTP round-trip dominates wall-clock time (100ms–30s depending on model and prompt length).
Pydantic validation is repeated on every invoke; cached schema parsing keeps it under 1ms for typical inputs.
Callback fan-out is synchronous by default — a slow handler will slow every call. Async tracers (LangSmith) push work to a background thread to avoid blocking.

Resource Usage

Memory: token-history buffers (in memory/) and document caches (in retrieval pipelines) are the largest steady-state allocations.
I/O: bounded by provider rate limits; vector store I/O for retrieval-flavored modules.
Concurrency: async support via a* methods; agents and parallel chains use asyncio.gather for concurrent tool calls.

Directory Map

Source tree for llms

llms/
├── __init__.py
├── ai21.py
├── aleph_alpha.py
├── amazon_api_gateway.py
├── anthropic.py
├── anyscale.py
├── arcee.py
├── aviary.py
├── azureml_endpoint.py
├── baidu_qianfan_endpoint.py
├── bananadev.py
├── base.py
├── baseten.py
├── beam.py
├── bedrock.py
├── bittensor.py
├── cerebriumai.py
├── chatglm.py
├── clarifai.py
├── cloudflare_workersai.py
├── cohere.py
├── ctransformers.py
├── ctranslate2.py
├── databricks.py
├── deepinfra.py
├── deepsparse.py
├── edenai.py
├── fake.py
├── fireworks.py
├── forefrontai.py
├── gigachat.py
├── google_palm.py
├── gooseai.py
├── gpt4all.py
├── gradient_ai.py
├── grammars/
├── huggingface_endpoint.py
├── huggingface_hub.py
├── huggingface_pipeline.py
├── huggingface_text_gen_inference.py
├── human.py
├── javelin_ai_gateway.py
├── koboldai.py
├── llamacpp.py
├── loading.py
├── manifest.py
├── minimax.py
├── mlflow.py
├── mlflow_ai_gateway.py
├── modal.py
├── mosaicml.py
├── nlpcloud.py
├── octoai_endpoint.py
├── ollama.py
├── opaqueprompts.py
├── openai.py
├── openllm.py
├── openlm.py
├── pai_eas_endpoint.py
├── petals.py
    ... (more files truncated)