storage (+ docstore) — Key-value byte stores: in-memory, file, Redis, encoder layers

Key-value byte stores: in-memory, file, Redis, encoder layers

Module langchain_classic/storage/ — Key-value byte stores: in-memory, file, Redis, encoder layers. Part of the State & Tools group.

🎯 Why This Module Exists

This module is documented from the storage_family_deep.md Phase 1B research artifact. See sections below for design decisions, algorithms, error philosophy, performance, and evolution clues.

Source files

Sub-directories

State & Tools

Architecture

flowchart LR
  subgraph Core["langchain_classic/storage/"]
    BASE["Base abstraction"]
    IMPLS["Concrete implementations"]
    BASE --> IMPLS
  end
  subgraph Up["Upstream callers"]
    U1["chains/"]
    U2["agents/"]
  end
  subgraph Down["Downstream"]
    D1["langchain_core"]
    D2["External providers"]
  end
  Up --> Core
  Core --> Down

Data Flow

sequenceDiagram
  participant Caller as Caller
  participant Storage as storage (+ docstore)
  participant Core as langchain_core
  participant Ext as External
  Caller->>Storage: invoke(input)
  Note over Storage: Validate & route
  Storage->>Core: delegate primitive
  Core->>Ext: provider call
  Ext-->>Core: response
  Core-->>Storage: typed output
  Storage-->>Caller: result

Existence rationale

This family handles document substrates: how raw content gets in (document_loaders), how it's chunked / filtered (document_transformers, text_splitter), where it's indexed (vectorstores), where its raw bytes live (storage, docstore), and how the substrate's interface looks to the rest of the library. Most concrete classes (PyPDFLoader, BeautifulSoupTransformer, Chroma, FAISS, Pinecone) live in langchain_community or provider packages; what langchain_classic keeps are the abstract contracts (BaseLoader, VectorStore, BaseDocumentTransformer, ByteStore, Docstore) plus a few non-trivial in-tree implementations (InMemoryDocstore, LocalFileStore, EmbeddingsRedundantFilter, LongContextReorder).

Design decisions

Decision	Choice	Alternative	Rationale
`Document = (page_content, metadata)`	Tuple-like Pydantic class	Untyped dict	Strong typing across the pipeline; metadata is canonical
Loaders return iterables	`lazy_load() -> Iterator[Document]`	List	Supports streaming huge corpora without loading all into RAM
Vectorstore as Retriever	`VectorStore.as_retriever()`	Caller wraps manually	One-line adapter; ensures the retriever uses the store's native MMR/filter capabilities
Splitters live on `Document`	`split_documents(docs) -> list[Document]`	String-only splitters	Carries metadata through (page numbers, source URIs) — critical for citations
`ByteStore` separate from `BaseStore`	`ByteStore` is `BaseStore[str, bytes]`	One typed store	Encoder-backed pattern wraps a ByteStore to give typed views
`LocalFileStore` over flat directory	One key-per-file	SQLite	Easy to inspect / rsync; trivial cross-process
`EncoderBackedStore` adapter	Wraps a ByteStore with `key_encoder/value_encoder/value_decoder`	Bake encoding into stores	Lets users use the same backend for raw bytes and typed values

Algorithm deep-dives

1. `RecursiveCharacterTextSplitter`

Problem. Naïve fixed-size chunking cuts mid-sentence and breaks code blocks. We want chunks that respect structure — paragraphs > sentences > words > characters — falling back as needed.

Trace. With separators ["\n\n", "\n", " ", ""]:

Try splitting on "\n\n". If a piece is still too big, recurse with the next separator.
Continue down the list until each piece ≤ chunk_size.
Re-merge adjacent pieces back up to chunk_size to maximise context.
Add chunk_overlap characters from the previous chunk to the next.

Why this works. Most documents have hierarchy — preserving it means summaries of chunks make sense individually. Overlap prevents losing context at boundaries.

2. `EmbeddingsRedundantFilter`

Problem. Retrievers often return near-duplicates from over-aggressive chunking. Drop them before they waste prompt tokens.

Trace.

Embed all returned docs once.
Greedy: for each doc in order, drop if its embedding's max cosine similarity to any already-kept doc exceeds similarity_threshold (default 0.95).
Return survivors.

Complexity. O(K²) similarity comparisons for K retrieved docs — K is usually ≤ 20.

3. `LongContextReorder` (Lost-in-the-Middle)

Problem. Empirical: LLMs attend best to the start and end of long contexts; the middle gets ignored. Reorder retrieved docs to put the most relevant at the edges.

Trace. Given docs sorted by relevance (descending):

Even-ranked docs go to the start (in order).
Odd-ranked docs go to the end (in reverse).
Result: [1st, 3rd, 5th, …, 4th, 2nd] — most relevant at extremes.

Why. Mitigates the U-shaped attention curve documented in the LiM paper.

Error philosophy

Loaders raise on fatal I/O (file not found) and emit warnings on partial failures (a corrupted page in a PDF skips that page rather than killing the load). Vectorstores propagate vendor errors. Splitters never error — at worst they emit a chunk larger than chunk_size if no separator can split it.

Performance characteristics

Loaders: I/O-bound; lazy iteration is the only protection against OOM.
Splitters: Pure CPU; tens of MB/s.
Embedding redundant filter: dominated by the embedding API (one call for the batch).
LongContextReorder: O(K), free.

Evolution clues

text_splitter.py is now a re-export of langchain_text_splitters (a separate package, used by both classic and community).
docstore/ is the oldest module — InMemoryDocstore predates ByteStore and persists for FAISS interop.
storage/_lc_store.py is the newest piece, tying ByteStore to LangChain's serialisation.

Step-by-Step Walkthrough

Follow a typical operation through the storage/ module:

Caller invokes the public surface — usually .invoke(input) on a Runnable, or a legacy __call__(inputs_dict) on a Chain subclass.
Pydantic validation — every entry point is a BaseModel; constructor args, input dicts, and tool args_schema are validated before any expensive work runs. Fail-fast on type errors.
CallbackManager.configure — a fresh manager is built per call, fans on_*_start events to all subscribers (LangSmith tracer, stdout handler, custom logging), and stamps a UUID4 run_id for tree reconstruction.
storage (+ docstore)-specific logic — the module's core method runs: prompt rendering, retrieval, tool execution, memory load, document split, etc. See the deep-dive sections above.
Provider call — typically a single HTTP request to OpenAI/Anthropic/etc., or a local SDK call. Retries are handled by tenacity with exponential backoff and a configurable max-attempts knob.
Streaming fan-out — if the provider streams tokens, each chunk fires on_llm_new_token through the callback manager so downstream observers (UI, log, tracer) see partial output.
OutputParser pass — if a parser is attached, the raw provider output is parsed into a structured Python object; OutputParserException may trigger a retry chain (RetryWithErrorOutputParser).
Memory write-back — for chains/agents that have a BaseMemory attached, save_context(inputs, outputs) persists the turn before returning to the caller.
on_*_end fan-out — the callback manager closes the run span; tracers serialise the Run tree and ship it to LangSmith asynchronously.
Result returned — caller receives a typed object (AIMessage, dict, BaseModel, list[Document], ...) matching the module's declared output schema.

What Could Go Wrong

Failure Mode	Trigger	Detection	Recovery
Provider 5xx / rate-limit	OpenAI/Anthropic/etc. transient error	HTTP status from SDK	tenacity retries with exponential backoff up to `max_retries`; then surface to caller
Output parser failure	LLM returned malformed JSON / wrong schema	`OutputParserException` in parser	`RetryWithErrorOutputParser` re-prompts with the parser error included; or surface
Pydantic validation	Caller passes wrong-shaped dict	`ValidationError` at `__init__` / `invoke`	Surface immediately — no fallback is correct here
Callback handler exception	User-supplied tracer raises	Try/except around each fan-out	Logged and swallowed by default; `raise_error=True` to opt in to propagation
Provider auth missing	`OPENAI_API_KEY` unset	SDK `AuthenticationError`	Surface with a helpful message; no retry (would just fail again)

Performance Profile

Hot Paths

Provider HTTP round-trip dominates wall-clock time (100ms–30s depending on model and prompt length).
Pydantic validation is repeated on every invoke; cached schema parsing keeps it under 1ms for typical inputs.
Callback fan-out is synchronous by default — a slow handler will slow every call. Async tracers (LangSmith) push work to a background thread to avoid blocking.

Resource Usage

Memory: token-history buffers (in memory/) and document caches (in retrieval pipelines) are the largest steady-state allocations.
I/O: bounded by provider rate limits; vector store I/O for retrieval-flavored modules.
Concurrency: async support via a* methods; agents and parallel chains use asyncio.gather for concurrent tool calls.

Directory Map

Source tree for storage (+ docstore)

storage/
├── __init__.py
├── _lc_store.py
├── encoder_backed.py
├── exceptions.py
├── file_system.py
├── in_memory.py
├── redis.py
└── upstash_redis.py