rag-from-scratch

flowchart TD
  subgraph Library["src/ — the library"]
    Doc["Document
(universal payload)"]
    Loaders["loaders/
BaseLoader, PDFLoader"]
    Splitters["text-splitters/
Base/Character/Recursive/Token"]
    Embeds["embeddings/
EmbeddingModel + EmbeddingCache"]
    LLMs["llms/
BaseLLM + LlamaCpp"]
    Chat["chat-models/
ChatLlamaCpp"]
    Stub1["chains/ retrievers/
vector-stores/ prompts/
(scaffolded stubs)"]
  end
  subgraph Curriculum["examples/ — the lessons"]
    Ex0["00 How RAG Works
(no embeddings)"]
    Ex1["01 LLM basics"]
    Ex2["02 Data Loading"]
    Ex3["03 Splitting"]
    Ex4["04 Embeddings"]
    Ex5["05 Vector Store
(3 sub-lessons)"]
    Ex6["06 Retrieval Strategies
(5 sub-lessons)"]
  end
  subgraph Helpers["helpers/ — CLI chrome"]
    Out["OutputHelper
(chalk + ora)"]
    Llp["LlamaPrompter
(used by 06)"]
    Op["OpenAIClient"]
  end
  Loaders -->|emits| Doc
  Splitters -->|emits| Doc
  Doc --> Embeds
  Embeds -->|number array 384d| Stub1
  LLMs --> Chat
  Ex2 -->|imports| Loaders
  Ex3 -->|imports| Doc
  Ex4 -->|reimplements then uses| Embeds
  Ex5 -->|imports + uses HNSW| Doc
  Ex6 -->|uses| LLMs
  Ex6 -->|uses| Llp
  Ex0 -->|standalone| Curriculum
  Curriculum -->|formatted via| Out

sequenceDiagram
    participant U as User
    participant PDF as PDFLoader
    participant Split as RecursiveCharacterTextSplitter
    participant Cache as EmbeddingCache
    participant EM as EmbeddingModel
    participant Llama as node-llama-cpp
    U->>PDF: new PDFLoader(url).load()
    Note over PDF: WHY: pdf-parse normalises
whitespace + page boundaries
    PDF-->>U: Document[] with metadata.page
    U->>Split: splitDocuments(docs)
    Note over Split: WHY: chunks must fit context window
and respect paragraph/sentence boundaries
    Split-->>U: Document[] (chunked, with overlap)
    U->>EM: embedDocuments(texts)
    EM->>Cache: get(text)
    alt Cache hit
        Cache-->>EM: vector
        Note over Cache,EM: WHY: embedding is the slow path
~10-50ms per call
    else Cache miss
        EM->>Llama: ctx.getEmbeddingFor(text)
        Llama-->>EM: Float32Array[384]
        EM->>Cache: set(text, vector)
    end
    EM-->>U: number[384][]

sequenceDiagram
    participant U as User Query
    participant LLM as LlamaPrompter
    participant Retr as retrieveParallel()
    participant DB as embedded-vector-db
    participant Fuse as rrfFuse(k=60)
    participant Gen as ChatLlamaCpp
    U->>LLM: "rewrite as 3 paraphrases"
    Note over LLM: WHY: single query misses
vocabulary mismatches
    LLM-->>U: [orig, para1, para2]
    U->>Retr: retrieveParallel(queries)
    par embed + search per query
        Retr->>DB: search(NS, vec1, k=5)
        Retr->>DB: search(NS, vec2, k=5)
        Retr->>DB: search(NS, vec3, k=5)
    end
    DB-->>Retr: 3 ranked lists
    Retr->>Fuse: rrfFuse(lists)
    Note over Fuse: WHY: similarity scores from
different queries aren't comparable
    Fuse-->>U: top-N fused
    U->>Gen: prompt + retrieved context
    Gen-->>U: grounded answer

flowchart LR
  L0["00 How RAG works
(keyword)"] --> L1["01 LLM basics"]
  L1 --> L2["02 Data loading"]
  L2 --> L3["03 Splitting"]
  L3 --> L4["04 Embeddings"]
  L4 --> L5a["05/01 In-memory store"]
  L5a --> L5b["05/02 NN search"]
  L5b --> L5c["05/03 Metadata filtering"]
  L5c --> L6a["06/01 Basic retrieval"]
  L6a --> L6b["06/02 Query preprocess"]
  L6b --> L6c["06/03 Hybrid search"]
  L6c --> L6d["06/04 Multi-query"]
  L6d --> L6e["06/05 Query rewriting"]
  L6e -.->|promotion frontier| Lib["src/retrievers/
(scaffold)"]