loaders/

Read source documents into a uniform Document shape

An abstract BaseLoader contract plus one concrete loader: PDFLoader. URL or filesystem; whole-document or per-page; metadata-rich. Other formats (HTML, JSON, Markdown) are stubs — the contract is defined, implementations come later.

Partial PDF mature URL + FS pdfjs-dist

classes implemented

~250

LOC

URL/FS

sources

split modes

📚 What is this & why does it exist?

loaders/ is the entry point of any RAG pipeline: the bridge from "a file on disk or a URL on the web" to "a JavaScript Document object with metadata". Without this layer, every lesson would re-implement PDF parsing inline. With it, swapping a PDF for an HTML page or a JSON dump is a one-line change at the top of a script.

💡 Real-world analogy

A loader is the airport baggage handler. You hand over your bag in whatever format (hard case, soft duffel, briefcase); it comes out the other side as a uniform tagged item with a barcode and a destination written on it. The BaseLoader contract is the standard tag format; PDFLoader is the handler trained for hard cases.

🎯 Why This Component

The interesting design move here is the uniform output. Every loader returns Document[] regardless of source. That single decision lets every downstream component (splitters, vector stores, retrievers) work without knowing or caring whether the data started as a PDF, a URL, or a JSON file. It's a small interface, but it's the keystone that makes the rest of the pipeline composable.

The PDFLoader is the only mature implementation. The others (HTML, JSON, MD, CSV, Web) are stubs — the lessons import them where useful but the implementations are deferred. This is honest: the project teaches RAG using PDFs (papers, books) which is the most common real-world source.

🏗️ Design Rationale

Decision	Choice	Why
Abstract base class	`BaseLoader` with abstract `load()` and concrete `loadAndSplit()`	`loadAndSplit` is the same logic for every loader (load → splitter.splitDocuments) — factor it out once.
URL vs filesystem	Auto-detect via `source.startsWith('http')`	One `load()` call works regardless. Failing this distinction would force callers to manually fetch + buffer.
Per-page mode	`splitPages: true` (default) emits one `Document` per PDF page	Page boundaries are the most useful citation unit for academic content. Lesson 02 uses this for "you can cite page 4".
Metadata enrichment	Load PDF info (Title, Author, etc.) once, attach to every page Document	Citations need the source title, not just a page number. Cheap to compute up front.
Custom `pageJoiner`	User-supplied function for joining pages into one big string	Default is `'\n\n'`. Code/tabular content might want different joining; the seam is there.
Stub other formats	Empty `index.js` placeholders	The contract is named; implementations land when needed. Honest about scope.

Architecture

classDiagram
    class BaseLoader {
      +source: string|Buffer
      +load()* Promise Document[]
      +loadAndSplit(splitter) Promise Document[]
    }
    class PDFLoader {
      +source: string|Buffer
      +splitPages: true
      +pageJoiner: '\n\n'
      +load() Promise Document[]
      +_fetchOrRead(source) Promise Buffer
      +_extractText(pdfDoc) Promise pages
      +_extractMetadata(pdfDoc)
    }
    class HTMLLoader { stub }
    class JSONLoader { stub }
    class MarkdownLoader { stub }
    class CSVLoader { stub }
    class WebLoader { stub }
    BaseLoader <|-- PDFLoader
    BaseLoader <|.. HTMLLoader
    BaseLoader <|.. JSONLoader
    BaseLoader <|.. MarkdownLoader
    BaseLoader <|.. CSVLoader
    BaseLoader <|.. WebLoader
    PDFLoader ..> Document : emits
    PDFLoader ..> pdfjs : uses getDocument
    BaseLoader ..> BaseTextSplitter : loadAndSplit

Lifecycle & Data Flow

sequenceDiagram
    participant U as User
    participant PL as PDFLoader
    participant Net as fetch / fs
    participant PJ as pdfjs-dist
    U->>PL: new PDFLoader(url, {splitPages:true})
    U->>PL: await load()
    alt URL
        PL->>Net: fetch(source)
        Net-->>PL: ArrayBuffer
    else filesystem
        PL->>Net: readFile(source)
        Net-->>PL: Buffer
    end
    PL->>PJ: getDocument({data: buffer}).promise
    PJ-->>PL: pdfDoc (numPages, getPage, getMetadata)
    loop for each page 1..numPages
        PL->>PJ: pdfDoc.getPage(i)
        PJ-->>PL: page
        PL->>PJ: page.getTextContent()
        PJ-->>PL: items[]
        PL->>PL: items.map(it=>it.str).join(' ')
    end
    PL->>PJ: pdfDoc.getMetadata()
    PJ-->>PL: {info: {Title, Author, ...}}
    alt splitPages
        PL->>PL: pages.map(p => new Document(p.text, {source, page:p.num, ...info}))
    else single doc
        PL->>PL: new Document(pages.map(p=>p.text).join(pageJoiner), {source, totalPages, ...info})
    end
    PL-->>U: Document[]

Interactions

From	To	Interface	Why this interaction exists
Lessons 02, 04, 05, 06	`PDFLoader`	`new PDFLoader(url).load()`	Every realistic-data lesson starts with a PDF.
`PDFLoader`	node:fetch / node:fs/promises	`fetch / readFile`	Source can be remote (URL) or local (path) — transparent to the caller.
`PDFLoader`	pdfjs-dist (Mozilla)	`getDocument / page.getTextContent / getMetadata`	The actual PDF parser. Battle-tested, supports modern PDFs.
`BaseLoader.loadAndSplit`	any text-splitter	`splitter.splitDocuments(docs)`	Convenience: one call to do the loader-then-splitter pattern.
`PDFLoader`	`Document`	`new Document(text, metadata)`	The uniform output type all downstream components consume.

📝 Annotated Code Walkthrough — `PDFLoader.load`

Resolve source to a buffer

async load() {
    const data = typeof this.source === 'string'
        ? await this._fetchOrRead(this.source)
        : this.source;

Three input types: URL string, filesystem path string, or already-loaded Buffer. The Buffer case lets users plug in custom transport (e.g., S3, sftp) without forking the loader.

Parse with pdfjs

    const pdfDoc = await getDocument({ data, useSystemFonts: true }).promise;
    const numPages = pdfDoc.numPages;

useSystemFonts:true avoids the missing-font warnings that pdfjs would otherwise spam to stderr. Cosmetic but important for clean lesson output.

Walk pages serially

    const pages = [];
    for (let i = 1; i <= numPages; i++) {
        const page = await pdfDoc.getPage(i);
        const content = await page.getTextContent();
        const text = content.items.map(it => it.str).join(' ');
        pages.push({ num: i, text });
    }

Serial, not parallel. pdfjs-dist isn't internally thread-safe across pages of a single document, and the per-page work is small. Parallelising here would be a bug, not a speedup.

Pull metadata

    const meta = await pdfDoc.getMetadata();
    const info = meta?.info || {};

Optional chaining because some PDFs lack the metadata stream entirely. Failing that, you'd get a Cannot read property 'info' of null.

Emit Documents

    if (this.splitPages) {
        return pages.map(p => new Document(p.text, {
            source: this.source, page: p.num, totalPages: numPages,
            title: info.Title, author: info.Author, ...info
        }));
    }
    const text = pages.map(p => p.text).join(this.pageJoiner);
    return [new Document(text, {
        source: this.source, totalPages: numPages, ...info
    })];
}

Per-page mode is the default because it preserves the most provenance. The whole-document mode is for use cases where chunk-level retrieval would be too granular — e.g., very short documents or when you want page-aware splitters to do the chunking themselves.

Deep-Dive Highlights

The URL/FS auto-detect Convenience

source.startsWith('http') is dumb-simple but correct for 99% of cases. Users who want HTTPS-only auth or custom headers can pre-fetch into a Buffer and pass that.

Page mode vs whole mode Tradeoff

Per-page wins when you want to cite (page N). Whole-document wins for very short PDFs where page boundaries are arbitrary.

Why the other formats are stubs Honesty

The README is upfront: this project teaches RAG, not data engineering. PDF + occasional URL covers the realistic exercises. The stubs document the intended API surface for future work.

⚠ Edge Cases & Boundaries

Scenario	What Actually Happens
PDF is image-only (scanned)	`getTextContent` returns empty items[]; pages have empty text. The lesson sees zero content — OCR is out of scope.
Network fetch fails	`fetch` throws; bubbles up to caller. No retry/backoff — lessons are interactive, user reruns.
PDF is encrypted	`getDocument` throws `PasswordException`. No password handling implemented.
Very large PDF (1000+ pages)	Serial page walk takes time but never spikes memory dangerously — pdfjs streams. The bottleneck moves to embedding.
Metadata field has weird Unicode	Spread `...info` propagates as-is into Document metadata. Downstream code receiving the metadata gets exactly what was in the PDF.

Directory Map

loaders/ files

src/loaders/
├── BaseLoader.js                ← abstract: load() + loadAndSplit()
├── PDFLoader.js                 ← URL/FS, splitPages, pdfjs-dist
├── HTMLLoader.js                ← stub (planned: cheerio)
├── JSONLoader.js                ← stub (planned: jq-like jsonpath)
├── MarkdownLoader.js            ← stub (planned: front-matter aware)
├── CSVLoader.js                 ← stub (planned: row-as-document)
├── WebLoader.js                 ← stub (planned: HTTP+sanitize)
└── index.js                     ← barrel export

Configuration Reference

Parameter	Default	Description
`PDFLoader.source`	(required)	URL string, filesystem path, or pre-loaded Buffer.
`PDFLoader.splitPages`	true	One Document per page when true; one Document for whole PDF when false.
`PDFLoader.pageJoiner`	`'\n\n'`	Used when splitPages is false.
`useSystemFonts` (passed to pdfjs)	true	Suppresses missing-font warnings.
Output metadata: `source`	(set)	Original source identifier.
Output metadata: `page`	(set when splitPages)	1-based page number.
Output metadata: `totalPages`	(set)	How many pages the PDF had.
Output metadata: `title / author`	(from PDF info)	From the embedded metadata stream.
`loadAndSplit(splitter)`	(method)	Inherited convenience for the load → split pattern.

Code Examples

Load a PDF from URL

import { PDFLoader } from './src/loaders/PDFLoader.js';

const docs = await new PDFLoader(
    'https://arxiv.org/pdf/2005.11401.pdf',
    { splitPages: true }
).load();

console.log(`${docs.length} pages from "${docs[0].metadata.title}"`);
console.log(docs[0].pageContent.slice(0, 200));

Local PDF + immediate splitting

import { PDFLoader, RecursiveCharacterTextSplitter } from './src/index.js';

const chunks = await new PDFLoader('./papers/rag.pdf').loadAndSplit(
    new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 200 })
);
// chunks carry both PDF metadata (page, source) and chunk metadata (chunk, totalChunks)

Pre-loaded Buffer (for custom transport)

import { readFile } from 'node:fs/promises';
import { PDFLoader } from './src/loaders/PDFLoader.js';

const buffer = await readFile('./encrypted-after-decrypt.pdf');
const docs = await new PDFLoader(buffer, { splitPages: false }).load();