embeddings/

Text → 384-d vectors with optional caching

Two classes (EmbeddingModel, EmbeddingCache) implementing LangChain-shaped embedQuery/embedDocuments over node-llama-cpp, with a SHA-256-keyed LRU cache as a side-channel that never breaks the primary path.

Mature 745 LOC SHA-256 keyed cache Cosine similarity

classes

745

LOC

384

default dims

10K

cache maxSize

batch size

📚 What is this & why does it exist?

embeddings/ turns text into fixed-length real-valued vectors so that "similar meaning" becomes "similar coordinates". The EmbeddingModel is a thin LangChain-shaped wrapper around node-llama-cpp's embedding context. The EmbeddingCache sits beside it as a transparent side-channel — cache hits return in microseconds, cache misses fall through to the model. Together they make repeat-query workloads (interactive chat, evaluation runs, debugging loops) fast without any complexity in the calling code.

💡 Real-world analogy

Imagine a translator who has a phrase book in their pocket. When you say something, they first check the phrase book; if your phrase is there, they read out the translation in a fraction of a second. If not, they translate from scratch and add the new phrase to the book. The book has a fixed number of pages, so when it fills up they tear out the oldest unused entry to make room. That's EmbeddingModel + EmbeddingCache — and the SHA-256 hash is just the phrase book's index, letting them flip to the right page in one move.

🎯 Why This Component

Embedding generation is the slow path of every RAG system. A single ctx.getEmbeddingFor(text) call routes through llama.cpp, runs a forward pass through bge-small-en-v1.5 (~33 M params), and returns a Float32Array. On a CPU laptop that takes 10–50 ms per call. In an interactive RAG loop where the same documents are re-embedded across runs (debug, eval, demo), that adds up to seconds of wasted compute. EmbeddingCache exists for exactly this scenario: amortise the expensive call across runs by keying on SHA-256(text) and persisting to disk.

The choice to make caching optional and side-channel is deliberate. Dropping the cache should require zero code change in the calling site — the user passes cache:false in the constructor and the embedQuery code path skips the lookup entirely. This pattern (configurable side-channel) is one of the cleanest abstractions in the project.

🏗️ Design Rationale

Decision	Choice	Why
Cache key	SHA-256(text)	Fixed-length keys (64 hex chars) regardless of input size; hides text in disk dumps; collision-resistant. Tradeoff: ~5µs per hash, dwarfed by the embedding call.
Cache eviction	LRU by linear scan over `Map.entries()`	Eviction is rare on the hot path; doubly-linked-list LRU (O(1)) would add 30+ lines of bookkeeping. Readability wins.
Cache injection	`EmbeddingModel` takes `cache:bool` and constructs `new EmbeddingCache()` internally	Simpler API than dependency injection. The user can still call `model.cache.saveToDisk()` directly.
Batch size 32	Default, override per `EmbeddingModel`	Balances Promise.all parallelism vs llama.cpp serialisation cost. Larger batches don't help (llama.cpp is single-threaded per context).
Float32Array → Array conversion	`Array.from(embedding.vector)` on every call	JSON-serialisable for cache persistence; standard Array methods work. Tradeoff: 2× memory but cache size is small.
Manual cosine in `similarity()`	Hand-written loop, no SIMD	Educational clarity. The reader sees `(a·b)/(‖a‖·‖b‖)`. Real RAG goes through the vector DB anyway.

Architecture

classDiagram
    class EmbeddingModel {
      -modelPath: string
      -dimensions: 384
      -batchSize: 32
      -useCache: bool
      -cache: EmbeddingCache?
      -llama / model / context
      -initialized: bool
      +constructor(config)
      +initialize() Promise
      +embedQuery(text) Promise number[]
      +embedDocuments(texts, onProgress) Promise number[][]
      +embedDocumentsWithMetadata(docs)
      +similarity(text1, text2) Promise number
      -_cosineSimilarity(v1, v2) number
      -_ensureInitialized()
      +saveCache(filepath)
      +loadCache(filepath)
      +cleanup()
    }
    class EmbeddingCache {
      -maxSize: 10000
      -useHash: true
      -hashAlgorithm: 'sha256'
      -cache: Map~string, Entry~
      -stats: Stats
      +get(text) Promise number[]?
      +set(text, vector) Promise
      +has(text) Promise bool
      +size() number
      +clear()
      +keys() string[]
      +getStats() Stats
      +getTopEntries(limit) Entry[]
      +saveToDisk(filepath) Promise
      +loadFromDisk(filepath) Promise
      -_generateKey(text) string
      -_evictLRU()
    }
    class Entry {
      vector: number[]
      text: string?
      timestamp: number
      lastAccessed: number
      hits: number
    }
    class Stats {
      hits: number
      misses: number
      sets: number
      evictions: number
    }
    EmbeddingModel *-- EmbeddingCache : optional
    EmbeddingCache *-- Entry
    EmbeddingCache *-- Stats

Lifecycle & Data Flow

sequenceDiagram
    participant U as User code
    participant EM as EmbeddingModel
    participant EC as EmbeddingCache
    participant Crypto as crypto.createHash
    participant LM as llama.context
    U->>EM: new EmbeddingModel({modelPath, cache:true})
    Note over EM: cache = new EmbeddingCache()
(no model loaded yet)
    U->>EM: await initialize()
    EM->>LM: getLlama() + loadModel + createEmbeddingContext
    LM-->>EM: context
    Note over EM: ~1.5s cold start
    U->>EM: embedQuery("hello world")
    EM->>EC: get("hello world")
    EC->>Crypto: createHash('sha256').update(text).digest('hex')
    Crypto-->>EC: 64-char hex
    alt key in Map
        EC->>EC: entry.lastAccessed = now()
entry.hits++
stats.hits++
        EC-->>EM: vector
        EM-->>U: number[384]
        Note over U: ~5µs total
    else key absent
        EC->>EC: stats.misses++
        EC-->>EM: null
        EM->>LM: ctx.getEmbeddingFor("hello world")
        LM-->>EM: Float32Array(384)
        EM->>EM: Array.from(vector)
        EM->>EC: set("hello world", vector)
        EC->>EC: if size >= maxSize: _evictLRU()
        EC->>EC: cache.set(key, {vector, lastAccessed, hits:0, ...})
stats.sets++
        EM-->>U: number[384]
        Note over U: ~10-50ms
    end

Interactions

From	To	Interface	Why this interaction exists
Lesson 04+	`EmbeddingModel`	`embedQuery / embedDocuments`	Every RAG pipeline starts by turning text into vectors.
`EmbeddingModel`	`EmbeddingCache`	`get / set`	Avoid re-embedding identical text across runs — the slow path.
`EmbeddingModel`	node-llama-cpp	`getLlama / loadModel / createEmbeddingContext / getEmbeddingFor`	The actual neural-network inference. Wrapped to keep the rest of `src/` agnostic of llama.cpp internals.
`EmbeddingCache`	node:crypto	`createHash('sha256').update(text).digest('hex')`	Fixed-length, collision-resistant cache key.
`EmbeddingCache`	node:fs/promises	`writeFile / readFile`	Persist cache between runs as JSON.
User code	`EmbeddingModel.similarity`	two-string convenience	Educational only — real RAG uses the vector DB. Useful for the similarity-basics lesson.
User code	`EmbeddingModel.cleanup`	release resources	llama.cpp holds an mmap'd model and a context; cleanup is good hygiene before exit.

📝 Annotated Code Walkthrough — `EmbeddingModel.embedDocuments`

The mature batch-embedding path. Every step exists for a reason — remove any one and a real workload breaks.

Guard against use-before-init

async embedDocuments(texts, onProgress = null) {
    this._ensureInitialized();

Throws synchronously if the user forgot await initialize(). Without this, the next line would NPE deep inside llama.cpp with an opaque error.

Reserve output buffer + progress counter

    const vectors = [];
    let processed = 0;

processed lives outside the loop because the inner map closure increments it. Without the closure, the progress callback would fire in batch-aligned bursts (boring) instead of per-document (informative).

Outer loop: walk in batch-sized strides

    for (let i = 0; i < texts.length; i += this.batchSize) {
        const batch = texts.slice(i, Math.min(i + this.batchSize, texts.length));

Why batch? Promise.all over the entire array would create thousands of pending promises — memory pressure plus llama.cpp serialises anyway. Batches of 32 give the GC a chance to breathe.

Parallel cache-or-compute per batch

        const batchVectors = await Promise.all(
            batch.map(async (text) => {
                if (this.cache) {
                    const cached = await this.cache.get(text);
                    if (cached) return cached;
                }
                const embedding = await this.context.getEmbeddingFor(text);
                const vector = Array.from(embedding.vector);
                if (this.cache) await this.cache.set(text, vector);
                processed++;
                if (onProgress) onProgress(processed, texts.length);
                return vector;
            })
        );

The cache check is per-text, not per-batch. A mostly-cached batch returns in microseconds without ever touching the model. A mostly-cold batch parallelises the model calls; llama.cpp serialises internally but Promise.all keeps the JS event loop responsive (progress callbacks fire promptly).

Concatenate and continue

        vectors.push(...batchVectors);
    }
    return vectors;
}

Spread-push because vectors.concat(batchVectors) would allocate a new array each iteration. Spread mutates in place — cheaper.

📝 Annotated Code Walkthrough — `EmbeddingCache._evictLRU`

Initialise tracking variables

_evictLRU() {
    let oldestKey = null;
    let oldestTime = Infinity;

Infinity as the sentinel ensures the first iteration always wins, regardless of actual timestamp values.

Linear scan

    for (const [key, entry] of this.cache.entries()) {
        if (entry.lastAccessed < oldestTime) {
            oldestTime = entry.lastAccessed;
            oldestKey = key;
        }
    }

O(n) per eviction. Why acceptable? Evictions only happen when cache.size >= maxSize AND the new key is missing — a rare condition once the cache is warm. The hot path is cache hits which are O(1) Map lookup.

Delete and tally

    if (oldestKey) {
        this.cache.delete(oldestKey);
        this.stats.evictions++;
    }
}

The if (oldestKey) guard handles the empty-cache case — though in practice _evictLRU is only called when size >= maxSize, so the cache is never empty here.

Deep-Dive Highlights

SHA-256 keying tradeoffs Mechanism

Why hash the text at all? Three reasons: fixed key length (saves Map memory for long texts), privacy in disk dumps, collision resistance. The cost (~5µs per hash) is invisible against the embedding call.

Promise.all batch parallelism Algorithm

Within a batch, every map callback fires immediately. llama.cpp serialises internally (single context), but the JS event loop stays responsive — progress callbacks fire as each completes, not in batch-aligned bursts.

Stats as a debugging aid Observability

getStats() returns hits, misses, hitRate%, evictions. The user can monitor cache effectiveness during a run; lessons 04/02 prints this at the end as a learning artefact.

⚠ Edge Cases & Boundaries

Scenario	What Actually Happens
User calls `embedQuery` before `initialize`	`_ensureInitialized()` throws `"EmbeddingModel not initialized. Call initialize() first."` — loud, fail-fast.
Cache disk file corrupted	`JSON.parse` throws inside `loadFromDisk`; caller catches, cache stays empty (cold start). Correctness intact, performance degraded.
Cache full at `maxSize=10000`	Each insert evicts the LRU entry. Stats track `evictions` for tuning. Working-set > cache size means thrashing.
Two texts hash to the same SHA-256	Collision — the second `set` overwrites the first. SHA-256 collisions are computationally infeasible at the scales involved here (10K entries).
Vector dimensions mismatch in `similarity()`	Throws `"Vectors must have same dimensions"` from `_cosineSimilarity`. Caller bug; not silently coerced.
Zero-magnitude vector (rare)	Cosine returns `0` instead of `NaN` (div-by-zero guard). Better than propagating NaN through the retrieval pipeline.

Directory Map

embeddings/ files

src/embeddings/
├── EmbeddingModel.js   (320 LOC)   ← The wrapper: initialize, embedQuery, embedDocuments, similarity, cache plumbing
├── EmbeddingCache.js   (425 LOC)   ← LRU cache with stats + JSON persistence
└── index.js                       ← Re-exports both classes

Configuration Reference

Parameter	Default	Description
`EmbeddingModel.modelPath`	(required)	Path to GGUF embedding model.
`EmbeddingModel.dimensions`	384	Expected embedding dimensions; must match model.
`EmbeddingModel.logLevel`	'error'	Passed to `getLlama({logLevel})`.
`EmbeddingModel.cache`	true	Toggle the cache side-channel entirely.
`EmbeddingModel.batchSize`	32	Per-batch `Promise.all` width.
`EmbeddingCache.maxSize`	10000	Eviction threshold.
`EmbeddingCache.useHash`	true	SHA-256 the text key.
`EmbeddingCache.hashAlgorithm`	'sha256'	Any `crypto.createHash` algo.
`onProgress(processed, total)`	null	Optional callback fired per text in batch loop.
`cleanup()`	(method)	Releases context/model/llama; idempotent.
`saveCache(filepath) / loadCache(filepath)`	(methods)	JSON dump + restore including stats and per-entry timestamps.
`getCacheStats()`	(method)	Snapshot of hit-rate, evictions, total requests.

Code Examples

Basic embedding with cache stats

import { EmbeddingModel } from './src/embeddings/EmbeddingModel.js';

const em = new EmbeddingModel({
    modelPath: './models/bge-small-en-v1.5.Q8_0.gguf',
    cache: true,
    batchSize: 32
});
await em.initialize();

const vec = await em.embedQuery('What is RAG?');
const docVecs = await em.embedDocuments([
    'RAG combines retrieval with generation.',
    'Embeddings turn text into vectors.',
    'The capital of France is Paris.'
], (n, total) => console.log(`${n}/${total}`));

console.log(em.getCacheStats());
// { enabled:true, size:4, maxSize:10000, hits:0, misses:4,
//   sets:4, evictions:0, hitRate:'0.00%', totalRequests:4 }

await em.cleanup();

Persist cache between runs

// First run
await em.embedDocuments(myDocs);
await em.saveCache('./.cache/embeddings.json');

// Second run (after restart)
await em.loadCache('./.cache/embeddings.json');
await em.embedDocuments(myDocs);  // mostly cache hits, ~10x faster
console.log(em.getCacheStats().hitRate);  // '99.50%'

Compare two texts directly

const score = await em.similarity(
    'machine learning is fun',
    'AI training is enjoyable'
);
console.log(score);  // ~0.78