evaluation — QA correctness, criteria, embedding distance, agent trajectory eval

QA correctness, criteria, embedding distance, agent trajectory eval

Module langchain_classic/evaluation/ — QA correctness, criteria, embedding distance, agent trajectory eval. Part of the Cross-cutting group.

🎯 Why This Module Exists

This module is documented from the evaluation_deep.md Phase 1B research artifact. See sections below for design decisions, algorithms, error philosophy, performance, and evolution clues.

Source files

Sub-directories

Cross-cutting

Architecture

flowchart LR
  subgraph Core["langchain_classic/evaluation/"]
    BASE["Base abstraction"]
    IMPLS["Concrete implementations"]
    BASE --> IMPLS
  end
  subgraph Up["Upstream callers"]
    U1["chains/"]
    U2["agents/"]
  end
  subgraph Down["Downstream"]
    D1["langchain_core"]
    D2["External providers"]
  end
  Up --> Core
  Core --> Down

Data Flow

sequenceDiagram
  participant Caller as Caller
  participant Evaluati as evaluation
  participant Core as langchain_core
  participant Ext as External
  Caller->>Evaluati: invoke(input)
  Note over Evaluati: Validate & route
  Evaluati->>Core: delegate primitive
  Core->>Ext: provider call
  Ext-->>Core: response
  Core-->>Evaluati: typed output
  Evaluati-->>Caller: result

Existence rationale

evaluation/ provides quantitative scoring of LLM outputs so users can answer "did my prompt/model change make things better?". It bundles ten flavours of evaluator — exact match, regex match, embedding distance, string distance, QA correctness, criteria (LLM judge), pairwise comparison, scoring, and trajectory eval (for agents) — under a uniform load_evaluator(EvaluatorType.X) API. Without it, every team would re-implement Levenshtein + cosine + LLM-as-judge from scratch.

Design decisions

Decision	Choice	Alternative	Rationale
Loader factory + enum	`load_evaluator(EvaluatorType.QA)`	Direct class instantiation	Single discoverable surface; supports lazy import of optional deps
`StringEvaluator` vs `PairwiseStringEvaluator` vs `AgentTrajectoryEvaluator` ABCs	Three contracts	One generic	Each takes different inputs (prediction vs A/B vs trajectory); separate ABCs prevent confusion
Criteria evaluator uses an LLM judge with CoT	LLM rates on `helpfulness`, `correctness`, etc., with chain-of-thought before score	Direct rating	CoT improves judge consistency (Anthropic eval studies); reduces "always 5/10" failure mode
Embedding distance pluggable	Cosine / Euclidean / Manhattan / Chebyshev / Hamming	Cosine only	Different embeddings calibrate differently; users pick
String distance via `rapidfuzz` (optional)	Lazy-import; raise if not installed	Bundle	Keeps the base install lightweight

Algorithm deep-dive — Criteria evaluator

Trace.

User picks a criterion ("helpfulness", or a custom dict).
The judge LLM is shown the input, the prediction, optionally the reference, and the criterion description.
Prompt asks for step-by-step reasoning then a Y/N verdict.
A specialized output parser extracts the verdict from the CoT.
Returns {"score": 1 if Y else 0, "value": Y/N, "reasoning": cot_text}.

Why CoT. Without it, judge LLMs collapse to "Y" most of the time; with it, the model has to articulate why, which exposes failure modes the user can act on.

Error philosophy

Loud failures, no silent zeros. A failed parser raises rather than returning 0 — silent zeros would corrupt aggregate metrics. Aggregation is the user's job (or LangSmith's).

Sub-modules at a glance

criteria/ — LLM judge with chain-of-thought
qa/ — exact-match and LLM-checked answer correctness
string_distance/ — Levenshtein, Jaro-Winkler, etc. via rapidfuzz
embedding_distance/ — cosine / euclidean / manhattan / chebyshev / hamming
comparison/ — pairwise A/B with LLM judge
scoring/ — Likert-scale scoring with reasoning
agents/ — TrajectoryEvalChain scores the entire agent trajectory (correctness + tool use)
parsing/ — checks parser output validity
regex_match/, exact_match/ — deterministic matchers

Step-by-Step Walkthrough

Follow a typical operation through the evaluation/ module:

Caller invokes the public surface — usually .invoke(input) on a Runnable, or a legacy __call__(inputs_dict) on a Chain subclass.
Pydantic validation — every entry point is a BaseModel; constructor args, input dicts, and tool args_schema are validated before any expensive work runs. Fail-fast on type errors.
CallbackManager.configure — a fresh manager is built per call, fans on_*_start events to all subscribers (LangSmith tracer, stdout handler, custom logging), and stamps a UUID4 run_id for tree reconstruction.
evaluation-specific logic — the module's core method runs: prompt rendering, retrieval, tool execution, memory load, document split, etc. See the deep-dive sections above.
Provider call — typically a single HTTP request to OpenAI/Anthropic/etc., or a local SDK call. Retries are handled by tenacity with exponential backoff and a configurable max-attempts knob.
Streaming fan-out — if the provider streams tokens, each chunk fires on_llm_new_token through the callback manager so downstream observers (UI, log, tracer) see partial output.
OutputParser pass — if a parser is attached, the raw provider output is parsed into a structured Python object; OutputParserException may trigger a retry chain (RetryWithErrorOutputParser).
Memory write-back — for chains/agents that have a BaseMemory attached, save_context(inputs, outputs) persists the turn before returning to the caller.
on_*_end fan-out — the callback manager closes the run span; tracers serialise the Run tree and ship it to LangSmith asynchronously.
Result returned — caller receives a typed object (AIMessage, dict, BaseModel, list[Document], ...) matching the module's declared output schema.

What Could Go Wrong

Failure Mode	Trigger	Detection	Recovery
Provider 5xx / rate-limit	OpenAI/Anthropic/etc. transient error	HTTP status from SDK	tenacity retries with exponential backoff up to `max_retries`; then surface to caller
Output parser failure	LLM returned malformed JSON / wrong schema	`OutputParserException` in parser	`RetryWithErrorOutputParser` re-prompts with the parser error included; or surface
Pydantic validation	Caller passes wrong-shaped dict	`ValidationError` at `__init__` / `invoke`	Surface immediately — no fallback is correct here
Callback handler exception	User-supplied tracer raises	Try/except around each fan-out	Logged and swallowed by default; `raise_error=True` to opt in to propagation
Provider auth missing	`OPENAI_API_KEY` unset	SDK `AuthenticationError`	Surface with a helpful message; no retry (would just fail again)

Performance Profile

Hot Paths

Provider HTTP round-trip dominates wall-clock time (100ms–30s depending on model and prompt length).
Pydantic validation is repeated on every invoke; cached schema parsing keeps it under 1ms for typical inputs.
Callback fan-out is synchronous by default — a slow handler will slow every call. Async tracers (LangSmith) push work to a background thread to avoid blocking.

Resource Usage

Memory: token-history buffers (in memory/) and document caches (in retrieval pipelines) are the largest steady-state allocations.
I/O: bounded by provider rate limits; vector store I/O for retrieval-flavored modules.
Concurrency: async support via a* methods; agents and parallel chains use asyncio.gather for concurrent tool calls.

Directory Map

Source tree for evaluation

evaluation/
├── __init__.py
├── agents/
├── comparison/
├── criteria/
├── embedding_distance/
├── exact_match/
├── loading.py
├── parsing/
├── qa/
├── regex_match/
├── schema.py
├── scoring/
└── string_distance/