RAGFlow

flowchart TD
  subgraph Frontend["Frontend Tier"]
    WEB["web/ React SPA"]
  end
  subgraph API_Tier["Python API Tier"]
    API["api/ Flask blueprints"]
    APIDB["api/db Peewee models + services"]
    AGENT["agent/ canvas + components"]
  end
  subgraph Workers["Worker Tier"]
    EXEC["rag/svr/task_executor.py"]
    PIPE["rag/flow/Pipeline DAG"]
  end
  subgraph Core["Core RAG Engine"]
    APP["rag/app per ParserType chunkers"]
    DEEP["deepdoc parsers + vision"]
    NLP["rag/nlp tokenizer + search"]
    LLM["rag/llm provider adapters"]
    GRAPH["rag/graphrag GraphRAG + Light"]
    RAPTOR["rag/raptor recursive index"]
  end
  subgraph Stores["Storage Backends"]
    DOCSTORE["ES / Infinity / OpenSearch / OB"]
    OBJSTORE["MinIO / S3 / Azure / GCS / OSS"]
    REDIS["Redis queue + locks + progress"]
    MYSQL["MySQL Tenant / KB / Doc / Task"]
  end
  subgraph GoTier["Parallel Go Tier"]
    GOSVR["internal/ Gin server"]
    CPP["internal/cpp tokenizer"]
  end
  subgraph Interop["Interop"]
    SDK["sdk/ Python wrapper"]
    MCP["mcp/ MCP server"]
    COMMON["common/ shared utils"]
    MEMORY["memory/ chat memory"]
  end

  WEB -->|REST + SSE| API
  WEB -->|REST| GOSVR
  API --> APIDB
  API --> AGENT
  APIDB -->|ORM| MYSQL
  API -->|enqueue| REDIS
  REDIS -->|consume| EXEC
  EXEC --> APP
  EXEC --> PIPE
  EXEC --> RAPTOR
  EXEC --> GRAPH
  PIPE -->|extends Graph| AGENT
  APP --> DEEP
  APP --> NLP
  APP --> LLM
  GRAPH --> LLM
  GRAPH --> NLP
  EXEC -->|index chunks| DOCSTORE
  EXEC -->|store files| OBJSTORE
  AGENT -->|retrieval| NLP
  AGENT -->|chat/embed| LLM
  API -->|search| NLP
  NLP -->|query| DOCSTORE
  GOSVR --> MYSQL
  GOSVR --> CPP
  GOSVR -->|same schema| DOCSTORE
  SDK -->|HTTP| API
  MCP -->|via SDK| SDK
  MEMORY -->|persists in| DOCSTORE
  COMMON -.->|imported by| API
  COMMON -.->|imported by| EXEC

sequenceDiagram
    participant U as User (web/)
    participant API as api/apps/document_app
    participant DB as MySQL (Document, Task)
    participant OBJ as MinIO/S3
    participant Q as Redis Queue
    participant W as rag/svr/task_executor
    participant DD as deepdoc parsers + vision
    participant CHK as rag/app chunker
    participant LLM as rag/llm/embedding
    participant DS as ES / Infinity
    U->>API: POST /v1/document/upload (multipart)
    API->>OBJ: write raw file
    Note over API,OBJ: Heavy bytes never touch the API process beyond a stream
    API->>DB: insert Document + Task rows
    API->>Q: LPUSH task_id
    API-->>U: 200 with task_id (returns fast)
    Note over U,API: API stays unblocked; ingest is async
    Q-->>W: BRPOP task_id
    W->>OBJ: read file
    W->>DD: parse layout + OCR if scanned
    Note over W,DD: deepdoc gives bbox-aware text + tables
    W->>CHK: chunk per ParserType (naive/paper/manual/...)
    CHK->>LLM: embed each chunk
    LLM-->>CHK: vectors
    CHK->>DS: bulk index chunks + vectors
    W->>DB: update Task progress (also pushes Redis log)
    Note over W,DB: progress visible to web/ via SSE polling

sequenceDiagram
    participant U as User (web/ chat)
    participant API as api/apps/restful_apis
    participant DLG as DialogService
    participant SRC as rag/nlp/search
    participant DS as ES / Infinity
    participant RR as rag/llm/rerank_model
    participant CHT as rag/llm/chat_model
    U->>API: POST /v1/conversation/completion (SSE)
    API->>DLG: build LLMBundle + retrieve
    DLG->>SRC: hybrid query (BM25 + dense)
    SRC->>DS: search content_ltks + vector
    DS-->>SRC: top-K candidates
    SRC->>RR: rerank top-K
    RR-->>SRC: scored top-N
    SRC-->>DLG: chunks + citations
    Note over DLG: Compose prompt with citations
    DLG->>CHT: stream completion
    CHT-->>DLG: token stream
    DLG-->>API: SSE token frames
    API-->>U: text/event-stream
    Note over API,U: Citations link back to original chunks for trust

sequenceDiagram
    participant U as User (web/ canvas)
    participant API as api/apps canvas
    participant G as agent.canvas.Graph
    participant N1 as Node Begin
    participant N2 as Node LLM or AgentWithTools
    participant N3 as Node Iteration or Switch
    participant T as agent/tools or rag/llm
    U->>API: run canvas (DSL JSON)
    API->>G: instantiate Graph(dsl)
    G->>N1: execute begin
    N1-->>G: initial variables
    G->>N2: schedule next ready node
    N2->>T: invoke LLM / tool / retrieval
    T-->>N2: result
    N2-->>G: write outputs to var scope
    Note over G,N2: scheduler picks edges by topological order + branch conditions
    G->>N3: evaluate switch/iteration
    N3-->>G: select next path or sub-graph
    Note over N3: iteration runs sub-graph per item with isolated scope
    G-->>API: SSE per-component progress
    API-->>U: live trace

sequenceDiagram
    participant Web as web/ SPA
    participant Py as Python api/ (Flask)
    participant Go as Go internal/ (Gin)
    participant DB as MySQL (shared)
    participant CPP as C++ rag_analyzer
    participant DS as Doc-store (shared)
    Web->>Py: heavy ops (upload, agent run, chat, GraphRAG build)
    Web->>Go: thin reads (search, list, fetch)
    Py->>DB: write tasks, docs, chats
    Go->>DB: read same schema
    Go->>CPP: cgo tokenize_query
    CPP-->>Go: terms + weights
    Go->>DS: search hybrid
    Py->>DS: index + search
    Note over Py,Go: They never call each other directly — coupling is via DB + doc-store