RAGFlow Wiki › API Module › Database Layer

📚 What is this & why does it exist?

Persistence layer using Peewee ORM with 15+ tables modeling tenants, users, knowledge bases, documents, and tasks. Implements multi-tenancy by always filtering on tenant_id, custom field types for JSON/list columns, and connection pooling with deadlock retry logic.

💡 Real-world analogy

Like a giant filing cabinet shared across many companies — every drawer is labeled with the tenant code, and every clerk who reaches in must verify they belong to that company before touching anything. The custom JSON fields are like sticky notes attached to each folder for free-form metadata.

Database Layer: Peewee ORM & Models

Peewee ORM models, database schema, custom field types, and multi-tenancy enforcement at the data access layer.

Analogy: The db/ layer is like a "filing system" in a company. Instead of storing papers in cabinets, it stores entity data (Users, Documents, Tasks, etc.) in a MySQL database. Each entity has a schema (columns, types, constraints). The Peewee ORM is the "librarian" who knows how to fetch, insert, and update records correctly. Services use the librarian to work with data, without needing to write SQL by hand.

Purpose & Architecture

The db/ sub-module contains:

Database Models Overview

RAGFlow's database schema includes 15+ Peewee models. Here are the core entities:

Model Class MySQL Table Purpose Key Fields
User user User accounts and authentication id, email, password_hash, tenant_id, created_at, is_active
Tenant tenant Multi-tenant org boundaries id, name, description, created_by (FK User), created_at, updated_at
Document document Uploaded documents (PDFs, Word, etc.) id, tenant_id, name, file_path, file_type, created_by (FK User), created_at, updated_at, deleted_at (soft delete)
Chunk chunk Document chunks (embeddings + text) id, document_id (FK Document), content, tokens, embedding (vector), created_at, deleted_at
Dialog dialog Conversation definitions id, tenant_id, name, description, created_by (FK User), created_at, updated_at
Conversation conversation Chat sessions within a dialog id, dialog_id (FK Dialog), user_id (FK User), created_at, updated_at
Message message Individual chat messages id, conversation_id (FK Conversation), role (user/assistant), content, created_at
Task task Background processing tasks id, tenant_id, document_id (FK Document), status (pending/processing/done/error), progress_json (JSONField), created_at, updated_at
KnowledgeBase knowledge_base Groupings of documents + chunks id, tenant_id, name, description, created_by (FK User), created_at, updated_at
Memory memory User memory/context for conversations id, user_id (FK User), content, created_at, updated_at

Schema Relationships & Foreign Keys

Key relationships in the schema:

Custom Peewee Field Types

RAGFlow defines custom Peewee field types to handle complex data structures:

JSONField

Stores JSON objects in TEXT columns. Used for semi-structured data (metadata, config, progress).

# Example in db_models.py
class Task(BaseModel):
    progress_json = JSONField(default={})
    
    # Usage in service
    task = Task.get(Task.id == task_id)
    task.progress_json = {'percent': 50, 'stage': 'embedding'}
    task.save()

ListField

Stores Python lists as JSON arrays. Useful for tags, keywords, etc.

class Document(BaseModel):
    tags = ListField(default=[])
    
    # Usage
    doc.tags = ['important', 'finance', 'q4-2024']
    doc.save()

SerializedField

Serializes Python objects using pickle or JSON. Used for complex objects that don't fit JSON.

class ConversationContext(BaseModel):
    context_obj = SerializedField()
    
    # Usage
    ctx = ConversationContext()
    ctx.context_obj = {'user_preferences': {...}, 'session_state': {...}}
    ctx.save()

Domain Services Layer

Services in db/services/ are the primary interface for business logic. Each service encapsulates one entity's behavior:

Service Class Module Responsibilities
DocumentService document_service.py CRUD for documents, file path resolution, soft delete, tenant isolation
ChunkService chunk_service.py Chunk creation/retrieval, embedding queries, vector similarity search
DialogService dialog_service.py Conversation template management, tenant-scoped queries
ConversationService conversation_service.py Chat session lifecycle, message coordination
MessageService message_service.py Message persistence, role-based filtering (user vs assistant)
TaskService task_service.py Background task lifecycle, status updates, progress tracking
KnowledgebaseService knowledgebase_service.py Knowledge base CRUD, document grouping, tenant isolation
LLMService llm_service.py LLM model configuration, credential management
MemoryService memory_service.py User context/memory management for conversations
FileService file_service.py File uploads to S3/blob storage, temp file cleanup

Joint Services for Cross-Domain Mutations

Some operations span multiple entities and must be atomic. Joint services coordinate these:

Multi-Tenancy Enforcement in Database Layer

Multi-tenancy is enforced at THREE levels:

  1. ORM Level: Peewee queries are auto-filtered by tenant_id where applicable. Services should use tenant_id in WHERE clauses.
  2. Service Level: Services validate tenant_id before mutations. Example: 
    def delete_document(self, doc_id, tenant_id):
    doc = Document.get_by_id(doc_id)
    if doc.tenant_id != tenant_id:
        raise TenantMismatch(f"Doc {doc_id} not in tenant {tenant_id}")
    doc.deleted_at = datetime.now()
    doc.save()
  3. API Layer: All handlers pass g.tenant_id to service calls. Never trust request parameters for tenant_id.

Connection Pooling & Performance

RAGFlow uses Peewee's connection pool (typically 10–20 connections) to handle concurrent requests:

Soft Deletes & Data Integrity

Most entities use soft deletes (a deleted_at timestamp column) instead of hard deletes:

# Example in db_models.py
class Document(BaseModel):
    name = CharField()
    deleted_at = DateTimeField(null=True, default=None)
    
# Soft delete in service
doc.deleted_at = datetime.now()
doc.save()

# Query only active documents
active_docs = Document.select().where(Document.deleted_at.is_null())

Benefits:

Error Handling & Constraints

Peewee raises exceptions for constraint violations. Services catch and convert to domain exceptions:

from peewee import IntegrityError, DoesNotExist

class DocumentService:
    def create_document(self, tenant_id, name):
        try:
            doc = Document.create(
                tenant_id=tenant_id,
                name=name,
                created_by=current_user_id
            )
            return doc
        except IntegrityError as e:
            if "UNIQUE" in str(e):
                raise DocumentNameExists(f"Document '{name}' already exists in tenant")
            raise InternalError(f"DB constraint violation: {e}")
        except Exception as e:
            logger.exception(f"Failed to create document: {e}")
            raise InternalError("Failed to create document")

Key Files & Modules

Design Decisions & Trade-offs

Decision Rationale Trade-off
Peewee ORM (not SQLAlchemy) Simpler query API, less boilerplate, better for rapid development Less mature than SQLAlchemy, fewer advanced features (eager loading, relationship configuration)
Soft deletes instead of hard delete Preserves audit trail, enables undo, avoids orphaned foreign keys Requires WHERE clauses filtering deleted_at.is_null(); uses extra storage
Custom field types (JSONField, ListField) Handles semi-structured data without separate tables; portable across MySQL/PostgreSQL Less efficient for querying (can't easily search within JSON); requires deserialization on fetch
Explicit tenant_id checks in services Prevents data leaks; makes tenant isolation explicit and auditable Boilerplate in every service; easy to forget a check
Connection pooling (10–20 conns) Prevents connection exhaustion; reuses connections for efficiency Pool contention at high concurrency; deadlock retry overhead

Security Implications

Future Enhancements