llms/

Generation: prompt → text, with chat-template support

Two LangChain-shaped abstractions (BaseLLM, BaseChatModel) plus llama.cpp implementations (LlamaCpp, ChatLlamaCpp) and small factory helpers in llama_cpp.js. The chat model uses node-llama-cpp's session API for proper template rendering across Llama-3.x / Qwen / Phi formats.

Mature ~430 LOC Stream + non-stream node-llama-cpp

classes

factories

~430

LOC

families supported

📚 What is this & why does it exist?

llms/ is the generation half of RAG. Once you've retrieved relevant context, you stuff it into a prompt and ask a language model to answer. LlamaCpp handles raw completion; ChatLlamaCpp handles role-tagged conversations with the model's native chat template (e.g., Llama-3's <|begin_of_text|><|start_header_id|>...). The two factories in llama_cpp.js make instantiation a one-liner.

💡 Real-world analogy

A chat model is a multilingual diplomat. You hand them a script in English ({"role":"user","content":...}) and they translate it into the formal protocol the host country (the LLM family) expects — the bowing, titles, and salutations of Llama-3's special tokens. The base completion model is a calligrapher: you hand them text, they extend it in the same hand. Both speak to the same printer (llama.cpp), just with different etiquette.

🎯 Why This Component

The hard problem this module hides is chat-template fragmentation. Llama-3, Qwen, Phi, and Gemma all use different special tokens to demarcate system/user/assistant turns. Hand-rolling a prompt for each family is fragile and error-prone — one missing <|eot_id|> and the model rambles forever or repeats the user input. node-llama-cpp's LlamaChatSession reads the template from the GGUF metadata and applies it correctly. ChatLlamaCpp wraps that session API behind LangChain's {role, content}[] message shape so callers don't need to know which family they're talking to.

🏗️ Design Rationale

Decision	Choice	Why
Two abstractions	`BaseLLM` for completion, `BaseChatModel` for conversation	The two have fundamentally different prompt formats. Conflating them (LangChain has tried) leaks abstractions.
Session API for chat	Use `LlamaChatSession` instead of hand-built prompts	The session reads the GGUF chat template metadata and applies model-specific special tokens. Robust across Llama/Qwen/Phi/Gemma.
Streaming via async generator	`async *stream()` yielding tokens	Idiomatic JS — consumers use `for await`. No backpressure plumbing needed.
Lazy initialization	`_ensureInitialized()` guards every public method	Models are expensive to load. Letting users construct without awaiting init keeps the constructor synchronous.
Factory functions	`createLlamaCpp()`, `createChatLlamaCpp()`	Hides the ".initialize()" two-step. `const llm = await createLlamaCpp({...})` reads naturally.
Dimensions/contextSize as constructor opts	Passed through to llama.cpp	Reproducibility — the lessons set known values rather than relying on GGUF defaults.

Architecture

classDiagram
    class BaseLLM {
      +invoke(input)*
      +stream(input)*
      +batch(inputs)
      +_ensureInitialized()
      +cleanup()
    }
    class BaseChatModel {
      +invoke(messages)*
      +stream(messages)*
      +batch(arrays)
      +_formatMessages(messages)*
      +_ensureInitialized()
    }
    class LlamaCpp {
      +modelPath, contextSize
      +temperature, topP, topK
      +maxTokens, repeatPenalty
      -llama, model, context, sequence
      +initialize() Promise
      +invoke(prompt) Promise string
      +stream(prompt) AsyncGenerator
      +cleanup()
    }
    class ChatLlamaCpp {
      +modelPath, contextSize, ...
      -llama, model, context, sequence, session
      +initialize() Promise
      +invoke(messages) Promise string
      +stream(messages) AsyncGenerator
      +_buildSession()
      +cleanup()
    }
    BaseLLM <|-- LlamaCpp
    BaseChatModel <|-- ChatLlamaCpp
    LlamaCpp ..> nodeLlamaCpp : completion API
    ChatLlamaCpp ..> nodeLlamaCpp : LlamaChatSession
    class Factories {
      createLlamaCpp(opts) Promise LlamaCpp
      createChatLlamaCpp(opts) Promise ChatLlamaCpp
    }
    Factories ..> LlamaCpp
    Factories ..> ChatLlamaCpp

Lifecycle & Data Flow — Chat Streaming

sequenceDiagram
    participant U as User
    participant F as createChatLlamaCpp
    participant CC as ChatLlamaCpp
    participant LCS as LlamaChatSession
    participant LM as llama.cpp
    U->>F: await createChatLlamaCpp({modelPath, ...})
    F->>CC: new ChatLlamaCpp({...})
    F->>CC: await initialize()
    CC->>LM: getLlama() + loadModel + createContext + getSequence
    CC->>LCS: new LlamaChatSession({contextSequence, systemPrompt})
    F-->>U: ready
    U->>CC: stream([{role:'user', content:'Q'}])
    CC->>LCS: promptWithMeta(content, {onToken})
    Note over LCS,LM: session formats with chat
template + sends to llama.cpp
    loop tokens emitted
        LM-->>LCS: token id
        LCS->>LCS: detokenize
        LCS-->>CC: onToken(text)
        CC-->>U: yield text
    end
    LCS-->>CC: final response
    Note over U: for-await loop ends naturally

Interactions

From	To	Interface	Why this interaction exists
Lessons 04, 05, 06	`ChatLlamaCpp`	`chat.invoke([{role:'user', content:...}])`	The generation step of every RAG lesson.
Examples (raw)	`LlamaCpp`	`llm.invoke(promptString)`	For lessons demonstrating raw completion (no chat template).
`ChatLlamaCpp`	`LlamaChatSession`	via `node-llama-cpp`	Applies model-specific chat template; tracks message history.
`ChatLlamaCpp.stream`	caller	`async *` generator yielding token strings	Streaming UX; consumer drives via `for await`.
`createLlamaCpp` / `createChatLlamaCpp`	respective classes	factory: construct + initialize + return	Hides the two-step instantiation.
Cleanup	llama.cpp resources	`session.dispose / context.dispose / model.dispose`	Releases mmap'd model. Prevents leaks across multiple instantiations in tests.

📝 Annotated Code Walkthrough — `ChatLlamaCpp.stream`

Guard initialization

async *stream(messages) {
    this._ensureInitialized();

Throws if the user constructed but didn't await initialize(). The factory hides this from the user; the manual path still has the safety net.

Extract latest user message + history

    const last = messages[messages.length - 1];
    if (last.role !== 'user') throw new Error('last message must be user');
    const history = messages.slice(0, -1);

The session API expects the next user prompt + system/historical context separately. Lessons construct messages as [{role:'system'}, {role:'user'}, {role:'assistant'}, {role:'user'}] — we split the tail off.

Sync session history

    if (history.length) {
        await this.session.setChatHistory(this._formatHistory(history));
    }

Re-priming the session lets the same instance handle independent conversations — useful in batch evaluation. Without this, the session would carry stale history across calls.

Stream via callback bridge

    const queue = [];
    let done = false;
    let resolveNext;
    const promise = this.session.promptWithMeta(last.content, {
        temperature: this.temperature,
        topP: this.topP,
        maxTokens: this.maxTokens,
        onToken: (tokens) => {
            const text = this.model.detokenize(tokens);
            queue.push(text);
            if (resolveNext) { resolveNext(); resolveNext = null; }
        }
    }).then(() => { done = true; if (resolveNext) resolveNext(); });

The bridge. node-llama-cpp's API is callback-based (onToken); the consumer-friendly contract is an async generator. We bridge by buffering tokens in queue[] and using a wake-up promise resolveNext to coordinate.

Drain loop

    while (!done || queue.length) {
        if (!queue.length) await new Promise(r => { resolveNext = r; });
        while (queue.length) yield queue.shift();
    }
    await promise;
}

Two-phase: wait for tokens, then drain everything queued. The final await promise surfaces any error from promptWithMeta; without it, errors during generation would be silently swallowed after the consumer's loop ended.

Deep-Dive Highlights

Callback-to-generator bridge Pattern

The onToken callback gets adapted into an async generator via a queue + wake-up promise. Any callback API can be lifted this way.

Why use the session API Robustness

GGUF embeds the chat template as metadata. LlamaChatSession reads it, so swapping Llama-3 for Qwen-2 is a model-path change, no code change.

Cleanup ordering Resource Mgmt

Dispose session before context before model. Wrong order can leave a zombie llama.cpp thread.

⚠ Edge Cases & Boundaries

Scenario	What Actually Happens
Last message is not user	Throws synchronously — the chat template only generates an assistant turn from a trailing user prompt.
Context overflow	llama.cpp truncates oldest tokens silently. Reduce `contextSize` or compress history; this module doesn't auto-summarise.
maxTokens exceeded	Generation stops mid-sentence. The streamed partial is what the consumer gets.
Model file missing	`loadModel` throws ENOENT; bubbles through `initialize()`.
Calling `invoke` while a stream is running	Sequence/session is single-threaded — second call blocks. Use one instance per concurrent stream.
Repeat penalty too high	Output devolves to disconnected tokens. Defaults are conservative (1.1).

Directory Map

llms/ files

src/llms/
├── BaseLLM.js              ← abstract: invoke, stream, batch
├── BaseChatModel.js        ← abstract: same surface, message[] inputs
├── LlamaCpp.js             ← concrete completion: model + context + sequence
├── ChatLlamaCpp.js         ← concrete chat: + LlamaChatSession + history sync
├── llama_cpp.js            ← createLlamaCpp + createChatLlamaCpp factories
└── index.js                ← barrel export

Configuration Reference

Parameter	Default	Description
`modelPath`	(required)	Path to GGUF model.
`contextSize`	4096	Token context window. Match to your prompt+answer budget.
`temperature`	0.7	Sampling temperature. 0 = greedy.
`topP`	0.95	Nucleus sampling threshold.
`topK`	40	Top-K sampling truncation.
`maxTokens`	512	Hard cap on generated tokens.
`repeatPenalty`	1.1	Discourages immediate repetition.
`systemPrompt` (chat only)	`null`	Initial system message.
`logLevel`	'error'	Suppress llama.cpp's debug spam.
`gpuLayers`	0	How many transformer layers to offload to GPU (if available).
`seed`	random	Reproducibility hook.

Code Examples

Streamed chat with system prompt

import { createChatLlamaCpp } from './src/llms/llama_cpp.js';

const chat = await createChatLlamaCpp({
    modelPath: './models/Llama-3.2-3B-Instruct-Q4_K_M.gguf',
    contextSize: 4096, temperature: 0.7, maxTokens: 512,
    systemPrompt: 'You are a precise technical assistant.'
});

for await (const chunk of chat.stream([
    { role: 'user', content: 'Explain RAG in one paragraph.' }
])) {
    process.stdout.write(chunk);
}
console.log();
await chat.cleanup();

Inside a RAG loop

const context = retrievedDocs.map(d => d.pageContent).join('\n\n');
const messages = [
    { role: 'system', content: 'Answer using only the provided context.' },
    { role: 'user', content: `Context:\n${context}\n\nQuestion: ${query}` }
];
const answer = await chat.invoke(messages);

Raw completion (non-chat)

import { createLlamaCpp } from './src/llms/llama_cpp.js';

const llm = await createLlamaCpp({ modelPath: './models/qwen2.5-1.5b.gguf', maxTokens: 100 });
console.log(await llm.invoke('Once upon a time, '));
await llm.cleanup();