text-splitters/

🎯 Why This Component

Splitting is non-obvious because the embedding model has zero awareness of chunk boundaries. If you slice mid-sentence, the chunk's embedding becomes ambiguous — representing half a thought. If you slice every paragraph regardless of size, you get embeddings that overflow the context window. The recursive separator approach navigates this with three concrete claims: (a) coarse separators preserve more meaning, so try them first; (b) when a piece is still too big, recurse to finer separators; (c) maintain chunkOverlap chars of context across boundaries so phrases near a cut are still indexed somewhere whole.

The component exists as a family rather than one class because the right strategy depends on the document. Code and CSV want a different separator priority than prose. Token-aware splitting (where the chunk-size budget is in tokens, not characters) wants a different length function. The base class factors out everything that's shared (overlap, document construction, metadata propagation) so each strategy is just an override of splitText().

🏗️ Design Rationale

Architecture

classDiagram
    class BaseTextSplitter {
      +chunkSize: 1000
      +chunkOverlap: 200
      +lengthFunction: fn
      +keepSeparator: false
      +splitText()*
      +splitDocuments(docs) Document[]
      +createDocuments(texts, metadatas) Document[]
      +mergeSplits(splits, separator) string[]
      +joinSplits(splits, sep) string
    }
    class CharacterTextSplitter {
      +separator: '\n\n'
      +splitText(text) string[]
    }
    class RecursiveCharacterTextSplitter {
      +separators: ['\n\n','\n','. ',' ','']
      +splitText(text) string[]
    }
    class TokenTextSplitter {
      +encodingName: 'cl100k_base'
      +lengthFunction: chars/4
      +splitText(text) string[]
    }
    BaseTextSplitter <|-- CharacterTextSplitter
    BaseTextSplitter <|-- RecursiveCharacterTextSplitter
    BaseTextSplitter <|-- TokenTextSplitter
    TokenTextSplitter ..> RecursiveCharacterTextSplitter : delegates to
    BaseTextSplitter ..> Document : emits via createDocuments

Lifecycle & Data Flow

sequenceDiagram
    participant U as User
    participant RS as RecursiveCharSplitter
    participant Base as BaseTextSplitter
    U->>RS: splitDocuments([Document(pageContent, meta)])
    RS->>Base: createDocuments([text], [meta])
    Base->>RS: splitText(text)
    Note over RS: pick coarsest separator
that exists in text
    RS->>RS: text.split(sep).filter(Boolean)
    loop for each split
        alt split <= chunkSize
            RS->>RS: temp.push(split)
        else split too big
            RS->>Base: mergeSplits(temp, sep)
            Base-->>RS: chunks (with overlap)
            alt has finer separators
                RS->>RS: recurse with nextSeparators
            else no finer
                RS->>RS: emit oversize as-is
            end
        end
    end
    RS->>Base: mergeSplits(remaining temp, sep)
    Base-->>RS: final chunks
    RS-->>Base: chunks: string[]
    Base->>Base: wrap each chunk in Document
add {chunk:j, totalChunks:N}
    Base-->>U: Document[]

Interactions

📝 Annotated Code Walkthrough — RecursiveCharacterTextSplitter.splitText

The most important algorithm in this module. Every line solves a specific failure mode.

splitText(text) {
    const finalChunks = [];
    let separator = this.separators.at(-1);
    let nextSeparators = [];

    for (let i = 0; i < this.separators.length; i++) {
        const sep = this.separators[i];
        if (text.includes(sep)) {
            separator = sep;
            nextSeparators = this.separators.slice(i + 1);
            break;
        }
    }

    const splits = text.split(separator).filter(Boolean);
    let temp = [];

    for (const s of splits) {
        if (this.lengthFunction(s) <= this.chunkSize) {
            temp.push(s);
        } else {

            if (temp.length) {
                finalChunks.push(...this.mergeSplits(temp, separator));
                temp = [];
            }

            if (nextSeparators.length === 0) {
                finalChunks.push(s);
            } else {
                const recursiveSplitter = new RecursiveCharacterTextSplitter({
                    ...this,
                    separators: nextSeparators
                });
                finalChunks.push(...recursiveSplitter.splitText(s));
            }
        }
    }

    if (temp.length) {
        finalChunks.push(...this.mergeSplits(temp, separator));
    }
    return finalChunks;
}

📝 BaseTextSplitter.mergeSplits — the overlap engine

mergeSplits(splits, separator) {
    const chunks = [];
    let current = [];
    let length = 0;

    for (const split of splits) {
        const splitLength = this.lengthFunction(split);
        const extraLength = current.length ? separator.length : 0;
        if (length + splitLength + extraLength > this.chunkSize) {

            if (current.length) {
                chunks.push(this.joinSplits(current, separator));
            }
            while (length > this.chunkOverlap && current.length) {
                length -= this.lengthFunction(current.shift()) + separator.length;
            }
        }

        current.push(split);
        length += splitLength + (current.length > 1 ? separator.length : 0);
    }

    if (current.length) {
        chunks.push(this.joinSplits(current, separator));
    }
    return chunks.filter(Boolean);
}

Deep-Dive Highlights

⚠ Edge Cases & Boundaries

Directory Map

src/text-splitters/
├── BaseTextSplitter.js              (99 LOC)   ← abstract: chunkSize/Overlap, mergeSplits, splitDocuments
├── CharacterTextSplitter.js         (18 LOC)   ← single-separator strategy
├── RecursiveCharacterTextSplitter.js (59 LOC)  ← priority-list recursion (recommended)
├── TokenTextSplitter.js             (23 LOC)   ← chars/4 heuristic + delegates to recursive
└── index.js                                    ← barrel re-export

Configuration Reference

Code Examples

Recursive splitter (recommended for prose)

import { RecursiveCharacterTextSplitter } from './src/text-splitters/index.js';

const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 800,
    chunkOverlap: 150,
    separators: ['\n\n', '\n', '. ', ' ', '']
});

const chunks = splitter.splitText(longDocument);
console.log(`${chunks.length} chunks`);
chunks.forEach((c, i) => console.log(`[${i}] ${c.slice(0, 80)}...`));

Split documents preserving metadata

import { PDFLoader, RecursiveCharacterTextSplitter } from './src/index.js';

const docs = await new PDFLoader(url, { splitPages: true }).load();
const chunks = await new RecursiveCharacterTextSplitter({
    chunkSize: 1000, chunkOverlap: 200
}).splitDocuments(docs);

// Each chunk carries: pageContent + {source, page, loc, chunk, totalChunks, ...}
const cited = chunks.map(c => `(p${c.metadata.page}, chunk ${c.metadata.chunk + 1}/${c.metadata.totalChunks}) ${c.pageContent}`);

Token-aware splitter

import { TokenTextSplitter } from './src/text-splitters/index.js';

// Approx: 1 token = 4 chars. So 256 "tokens" = 1024 chars in the budget.
const splitter = new TokenTextSplitter({ chunkSize: 256, chunkOverlap: 32 });
const chunks = splitter.splitText(longDocument);