🚀 Deployment & Scripts

API Server, Web Demo, and Model Conversion

Files

Deploy Modes

~510

Lines

🐍

Python

🔵 FastAPI · OpenAI API 🟢 Streamlit · WebUI 🟡 Model Conversion 🔴 Streaming

🏪 Bakery Storefront Analogy

Think of the scripts module as the storefront and packaging department of a bakery. The model (bread) has been baked in the training kitchen — now it needs to be displayed and served. serve_openai_api.py is the counter where customers (clients) place orders via a standard menu (OpenAI API format). web_demo.py is the cozy café area with a tasting counter. convert_model.py is the packaging station — slicing, boxing, and labeling products for different distribution channels (HuggingFace, Llama format). And chat_openai_api.py is the mystery shopper testing the service.

Architecture Overview

flowchart LR
    W["Model Weights\n(.pth / .safetensors)"] --> S["serve_openai_api.py\nFastAPI :8998"]
    W --> WD["web_demo.py\nStreamlit UI"]
    W --> C["convert_model.py\nFormat Converter"]

    S -->|"/v1/chat/completions"| CL["chat_openai_api.py\nTest Client"]
    S -->|"OpenAI API"| EXT["External Clients\ncurl / SDKs"]
    WD -->|"Local Mode"| WD
    WD -->|"API Mode"| S

    C --> HF["HuggingFace Format"]
    C --> LL["LlamaForCausalLM"]
    C --> NT["Native .pth (fp16)"]

    style S fill:#1f6feb,stroke:#58a6ff,color:#fff
    style WD fill:#238636,stroke:#3fb950,color:#fff
    style C fill:#9a6700,stroke:#d29922,color:#fff
    style CL fill:#da3633,stroke:#f85149,color:#fff

Key Functions & Classes

Name	Type	File	Description
`ChatRequest`	Pydantic Model	serve_openai_api.py	Request schema: model, messages, temperature, top_p, max_tokens, stream, tools
`CustomStreamer`	Class	serve_openai_api.py	Queue-based `TextStreamer` subclass for SSE streaming responses
`/v1/chat/completions`	Endpoint	serve_openai_api.py	OpenAI-compatible chat completion with streaming & non-streaming support
`process_assistant_content`	Function	web_demo.py	Renders `<think>`/`<answer>` tags as collapsible HTML details elements
`TextIteratorStreamer`	Class (ext)	web_demo.py	Token-by-token streamer for real-time chat display in Streamlit
`MODEL_PATHS`	Dict	web_demo.py	Maps model names (MiniMind2-R1, Small-R1, MoE, etc.) to weight paths
`convert_torch2transformers_minimind`	Function	convert_model.py	Native MiniMind `.pth` → HuggingFace Transformers format
`convert_torch2transformers_llama`	Function	convert_model.py	Converts to `LlamaForCausalLM` for broad ecosystem compatibility
`convert_transformers2torch`	Function	convert_model.py	Reverse conversion: HuggingFace → native `.pth` (half precision)
`chat_openai_api.py`	Script	chat_openai_api.py	OpenAI SDK client for testing; supports streaming & maintains history

API Request Flow

sequenceDiagram
    participant Client as Client / WebUI
    participant API as FastAPI Server
    participant Streamer as CustomStreamer
    participant Model as MiniMind Model
    participant LoRA as LoRA Weights (opt)

    Client->>API: POST /v1/chat/completions
    API->>API: Parse ChatRequest
    API->>Model: Load base weights
    API-->>LoRA: Merge LoRA (if present)

    alt Streaming Mode
        API->>Streamer: Create queue-based streamer
        API->>Model: model.generate(streamer=...)
        loop Token by token
            Model->>Streamer: put(token)
            Streamer->>API: yield from queue
            API->>Client: SSE data chunk
        end
        API->>Client: SSE [DONE]
    else Non-Streaming
        API->>Model: model.generate()
        Model->>API: full response
        API->>Client: JSON response
    end

Interactions & Dependencies

Component	Depends On	Interaction
`serve_openai_api.py`	FastAPI, uvicorn, transformers	Launches HTTP server on port 8998 with OpenAI-compatible routes
`serve_openai_api.py`	MiniMind model, tokenizer	Loads model weights and optional LoRA adapter at startup
`CustomStreamer`	TextStreamer (transformers)	Overrides `put()` to enqueue tokens for SSE streaming
`web_demo.py`	Streamlit, transformers	Provides interactive UI; can load model locally or call API endpoint
`web_demo.py`	serve_openai_api.py (API mode)	Proxies chat requests to FastAPI server when in API mode
`convert_model.py`	torch, transformers, LlamaConfig	Bidirectional conversion between native and HuggingFace formats
`chat_openai_api.py`	openai SDK	Tests the serve endpoint with streaming/non-streaming requests
All scripts	model/ module	Import model architecture classes for weight loading and inference

Deep-Dive Topics

🔌 OpenAI API Compatibility

How serve_openai_api.py mirrors the OpenAI /v1/chat/completions schema — request validation, response formatting, tool calling support, and drop-in SDK compatibility.

📡 Streaming Architecture

The CustomStreamer queue pattern: how tokens flow from model.generate() through a thread-safe queue into SSE chunks, enabling real-time token-by-token responses.

🔄 Model Format Conversion

Three-way conversion between native MiniMind .pth, HuggingFace Transformers, and LlamaForCausalLM formats — weight mapping, precision handling, and config generation.

Directory Map

scripts/
├── serve_openai_api.py    # ~178 lines — FastAPI server, OpenAI-compatible API
│   ├── ChatRequest          # Pydantic request model
│   ├── CustomStreamer        # Queue-based SSE streaming
│   └── /v1/chat/completions # Main endpoint
├── web_demo.py           # ~329 lines — Streamlit interactive chat UI
│   ├── MODEL_PATHS          # 5 model variants
│   └── process_assistant_content # Think/answer rendering
├── convert_model.py      # ~78 lines — Format converters
│   ├── torch → transformers # MiniMind & Llama formats
│   └── transformers → torch # Reverse, fp16
└── chat_openai_api.py    # ~34 lines — API test client

Configuration Reference

Parameter	Location	Default	Description
`port`	serve_openai_api.py	8998	HTTP server port for the FastAPI application
`temperature`	ChatRequest	0.7	Sampling temperature for response generation
`top_p`	ChatRequest	0.9	Nucleus sampling probability threshold
`max_tokens`	ChatRequest	8192	Maximum number of tokens in the generated response
`stream`	ChatRequest	false	Enable SSE streaming mode for token-by-token output
`tools`	ChatRequest	None	Optional tool/function definitions for function calling
`history_chat_num`	web_demo.py	5	Number of previous conversation turns to include in context
`max_new_tokens`	web_demo.py	1024	Maximum generation length in the web UI
`MODEL_PATHS`	web_demo.py	5 variants	MiniMind2-R1, MiniMind2-Small-R1, MiniMind2, MiniMind2-MoE, MiniMind2-Small
`use_lora`	serve_openai_api.py	False	Whether to load and merge LoRA adapter weights on top of base model

Code Examples

Starting the API Server

Bash — Launch OpenAI-compatible server

# Start the MiniMind API server on default port 8998
python scripts/serve_openai_api.py

# Test with curl
curl -X POST http://localhost:8998/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMind2",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true,
    "temperature": 0.7
  }'

Model Format Conversion

Python — Convert between model formats

from scripts.convert_model import (
    convert_torch2transformers_minimind,
    convert_torch2transformers_llama,
    convert_transformers2torch,
)

# Native MiniMind → HuggingFace Transformers
convert_torch2transformers_minimind(
    input_path="weights/minimind2.pth",
    output_dir="weights/hf_minimind2"
)

# Native → LlamaForCausalLM (ecosystem compatible)
convert_torch2transformers_llama(
    input_path="weights/minimind2.pth",
    output_dir="weights/llama_minimind2"
)

# HuggingFace → Native .pth (fp16)
convert_transformers2torch(
    input_dir="weights/hf_minimind2",
    output_path="weights/minimind2_fp16.pth"
)

Using the Chat Test Client

Python — Streaming chat client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8998/v1", api_key="none")

response = client.chat.completions.create(
    model="MiniMind2",
    messages=[{"role": "user", "content": "Explain transformers briefly"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)