document_transformers — Splitters, doc compressors, EmbeddingsRedundantFilter

Splitters, doc compressors, EmbeddingsRedundantFilter

Module langchain_classic/document_transformers/ — Splitters, doc compressors, EmbeddingsRedundantFilter. Part of the Retrieval & Data group.

🎯 Why This Module Exists

Transformers post-process Document lists between loading and embedding. Examples: RecursiveCharacterTextSplitter (split by paragraph then sentence then character), MarkdownHeaderTextSplitter (preserve heading hierarchy in metadata), EmbeddingsRedundantFilter (drop near-duplicate docs), LongContextReorder (move most-relevant docs to the ends, where LLMs attend best).

Source files

Sub-directories

Retrieval & Data

Architecture

flowchart LR
  subgraph Core["langchain_classic/document_transformers/"]
    BASE["Base abstraction"]
    IMPLS["Concrete implementations"]
    BASE --> IMPLS
  end
  subgraph Up["Upstream callers"]
    U1["chains/"]
    U2["agents/"]
  end
  subgraph Down["Downstream"]
    D1["langchain_core"]
    D2["External providers"]
  end
  Up --> Core
  Core --> Down

Data Flow

sequenceDiagram
  participant Caller as Caller
  participant Document as document_transformers
  participant Core as langchain_core
  participant Ext as External
  Caller->>Document: invoke(input)
  Note over Document: Validate & route
  Document->>Core: delegate primitive
  Core->>Ext: provider call
  Ext-->>Core: response
  Core-->>Document: typed output
  Document-->>Caller: result

Design Decisions

Decision	Choice Made	Alternatives	Why This Approach
Splitter as transformer	BaseDocumentTransformer interface	Bake into loader	Splitting strategy varies by use case more than by source
Recursive splitter	Try larger separators first, fall back to smaller	Fixed-size chunks	Preserves semantic boundaries
Long-context reorder	Reorder by relevance to combat the lost-in-the-middle effect	Trust LLM attention	Empirically retrieval quality drops for middle-of-context docs

Used inside ParentDocumentRetriever and ContextualCompressionRetriever to shape what reaches the LLM.

Step-by-Step Walkthrough

Follow a typical operation through the document_transformers/ module:

Caller invokes the public surface — usually .invoke(input) on a Runnable, or a legacy __call__(inputs_dict) on a Chain subclass.
Pydantic validation — every entry point is a BaseModel; constructor args, input dicts, and tool args_schema are validated before any expensive work runs. Fail-fast on type errors.
CallbackManager.configure — a fresh manager is built per call, fans on_*_start events to all subscribers (LangSmith tracer, stdout handler, custom logging), and stamps a UUID4 run_id for tree reconstruction.
document_transformers-specific logic — the module's core method runs: prompt rendering, retrieval, tool execution, memory load, document split, etc. See the deep-dive sections above.
Provider call — typically a single HTTP request to OpenAI/Anthropic/etc., or a local SDK call. Retries are handled by tenacity with exponential backoff and a configurable max-attempts knob.
Streaming fan-out — if the provider streams tokens, each chunk fires on_llm_new_token through the callback manager so downstream observers (UI, log, tracer) see partial output.
OutputParser pass — if a parser is attached, the raw provider output is parsed into a structured Python object; OutputParserException may trigger a retry chain (RetryWithErrorOutputParser).
Memory write-back — for chains/agents that have a BaseMemory attached, save_context(inputs, outputs) persists the turn before returning to the caller.
on_*_end fan-out — the callback manager closes the run span; tracers serialise the Run tree and ship it to LangSmith asynchronously.
Result returned — caller receives a typed object (AIMessage, dict, BaseModel, list[Document], ...) matching the module's declared output schema.

What Could Go Wrong

Failure Mode	Trigger	Detection	Recovery
Provider 5xx / rate-limit	OpenAI/Anthropic/etc. transient error	HTTP status from SDK	tenacity retries with exponential backoff up to `max_retries`; then surface to caller
Output parser failure	LLM returned malformed JSON / wrong schema	`OutputParserException` in parser	`RetryWithErrorOutputParser` re-prompts with the parser error included; or surface
Pydantic validation	Caller passes wrong-shaped dict	`ValidationError` at `__init__` / `invoke`	Surface immediately — no fallback is correct here
Callback handler exception	User-supplied tracer raises	Try/except around each fan-out	Logged and swallowed by default; `raise_error=True` to opt in to propagation
Provider auth missing	`OPENAI_API_KEY` unset	SDK `AuthenticationError`	Surface with a helpful message; no retry (would just fail again)

Performance Profile

Hot Paths

Provider HTTP round-trip dominates wall-clock time (100ms–30s depending on model and prompt length).
Pydantic validation is repeated on every invoke; cached schema parsing keeps it under 1ms for typical inputs.
Callback fan-out is synchronous by default — a slow handler will slow every call. Async tracers (LangSmith) push work to a background thread to avoid blocking.

Resource Usage

Memory: token-history buffers (in memory/) and document caches (in retrieval pipelines) are the largest steady-state allocations.
I/O: bounded by provider rate limits; vector store I/O for retrieval-flavored modules.
Concurrency: async support via a* methods; agents and parallel chains use asyncio.gather for concurrent tool calls.

Directory Map

Source tree for document_transformers

document_transformers/
├── __init__.py
├── beautiful_soup_transformer.py
├── doctran_text_extract.py
├── doctran_text_qa.py
├── doctran_text_translate.py
├── embeddings_redundant_filter.py
├── google_translate.py
├── html2text.py
├── long_context_reorder.py
├── nuclia_text_transform.py
├── openai_functions.py
└── xsl/