— Storage Backends & Abstraction

Multi-provider storage with encryption and compression

Storage factory, 10+ backends: ES, Infinity, OpenSearch, OB, MinIO, S3, Azure, GCS, OSS, OpenDAL, encrypted.

Classes

Files

~N k

Lines

Key Metric

❓ What is this & why?

The utils module abstracts storage behind a factory pattern. Users can deploy to S3, MinIO, Azure, GCS, or on-prem OpenDAL without changing code. Encryption and compression are pluggable features. This prevents vendor lock-in.

💡 Real-world analogy

Like a portable USB drive interface. The drive works with Windows, Mac, or Linux without knowing which OS it's connected to. Similarly, storage abstraction lets the RAG module work with any backend.

🤔 Why Is It Designed This Way?

{{DESIGN_RATIONALE}} — 150+ words explaining the design choices: what pattern does this follow and why? What constraints shaped the implementation? What trade-offs were made and what was sacrificed?

Design Choice	What Was Chosen	What Was Rejected	Why
Example: data layout	Array of structs	Struct of arrays	Random access pattern dominates; cache locality for single items matters more than SIMD sweeps

Architecture

flowchart TD
    subgraph API["Storage API"]
        A["StorageFactory"]
    end
    subgraph Backends["10+ Backends"]
        B1["S3"]
        B2["MinIO"]
        B3["Azure"]
        B4["GCS"]
        B5["Others..."]
    end
    subgraph Features["Features"]
        F1["Encryption"]
        F2["Compression"]
    end
    API -->|Route| Backends
    Backends --> Features

Key Classes

Class / Function	File	Purpose
`ClassName`	file.h	Brief description

Data Structures

Type	Name	Description
enum	`EnumName`	Brief description

Interactions

Direction	Partner	Interface	Description
→	Other Module	`methodName()`	Brief description

Configuration

Knob / Parameter	Default	Description
`param_name`	value	What it controls

Directory Map

Source tree for utils

submod_name/
├── file_a.ext                 ← Role: brief annotation
├── file_b.ext                 ← Role: brief annotation
├── file_c.ext                 ← Role: brief annotation
└── index.ext                  ← Role: entry point

📑 Annotated Code Walkthrough

The most important code in this sub-module, annotated with WHY each piece exists:

// Replace with actual code from the source
void key_function() {
    // ← WHY: Explain the purpose of this line
    // ← WHY: What constraint or requirement drives this?
}

Line-by-line:

Line N: What this does and WHY — what would break if removed
Line M: The design choice here — why this approach over the obvious alternative

⚠️ Edge Cases & Boundaries

Edge Case	What Happens	Where in Code	Risk Level
Empty input	How it's handled	function_name:line	🟢 Low
Concurrent access	How it's handled	function_name:line	🔴 High

Deep-Dive Highlights

Core mechanisms from this sub-module have been extracted into dedicated focus pages for in-depth exploration. Each page covers one algorithm, protocol, or subsystem with diagrams, code walks, and configuration details.

Deep-Dive Pages

Storage Backend Selection

sequenceDiagram
    participant App as Application
    participant Factory as StorageFactory
    participant S3 as S3
    participant MinIO as MinIO
    App->>Factory: create("s3")
    Factory->>S3: Initialize
    S3->>App: S3 instance
    App->>App: Store chunks
    App->>Factory: create("minio")
    Factory->>MinIO: Initialize
    MinIO->>App: MinIO instance

Code Examples

Example: Typical Usage

// Replace with actual code example
void example() {
    // ...
}