RAG/nlp/ — Tokenization & Query Analysis
CJK-aware tokenization, query expansion, term weighting
DART tokenizer (Rust FFI), search.py, query.py, term_weight.py, synonym expansion.
❓ What is this & why?
The nlp module handles tokenization, term weighting, and query expansion. It bridges raw text and retrieval indices. DART (Rust FFI) provides CJK-aware segmentation crucial for Asian-language support. Synonym expansion ensures queries like 'ML' match 'machine learning' documents.
💡 Real-world analogy
Like a librarian's indexing system. Before you can find books, the librarian must understand which words are equivalent ('ML' = 'machine learning'), how to parse titles correctly (especially in Asian languages), and which terms are more important than others. The NLP module does this automated understanding.
🤔 Why Is It Designed This Way?
{{DESIGN_RATIONALE}} — 150+ words explaining the design choices: what pattern does this follow and why? What constraints shaped the implementation? What trade-offs were made and what was sacrificed?
| Design Choice | What Was Chosen | What Was Rejected | Why |
|---|---|---|---|
| Example: data layout | Array of structs | Struct of arrays | Random access pattern dominates; cache locality for single items matters more than SIMD sweeps |
Contents
Architecture
flowchart LR
subgraph Tokenize["Tokenization"]
T1["DART Trie (CJK)"]
T2["Standard Tokenizer"]
end
subgraph Analyze["Query Analysis"]
A1["Term Weighting"]
A2["Synonym Expansion"]
end
subgraph Search["Search Preparation"]
S1["Query Enrichment"]
end
Tokenize --> Analyze
Analyze --> Search
Key Classes
| Class / Function | File | Purpose |
|---|---|---|
ClassName | file.h | Brief description |
Data Structures
| Type | Name | Description |
|---|---|---|
| enum | EnumName | Brief description |
Interactions
| Direction | Partner | Interface | Description |
|---|---|---|---|
| → | Other Module | methodName() | Brief description |
Configuration
| Knob / Parameter | Default | Description |
|---|---|---|
param_name | value | What it controls |
Directory Map
Source tree for nlp
submod_name/
├── file_a.ext ← Role: brief annotation
├── file_b.ext ← Role: brief annotation
├── file_c.ext ← Role: brief annotation
└── index.ext ← Role: entry point
📑 Annotated Code Walkthrough
The most important code in this sub-module, annotated with WHY each piece exists:
{{KEY_FUNCTION_NAME}}
// Replace with actual code from the source
void key_function() {
// ← WHY: Explain the purpose of this line
// ← WHY: What constraint or requirement drives this?
}Line-by-line:
- Line N: What this does and WHY — what would break if removed
- Line M: The design choice here — why this approach over the obvious alternative
⚠️ Edge Cases & Boundaries
| Edge Case | What Happens | Where in Code | Risk Level |
|---|---|---|---|
| Empty input | How it's handled | function_name:line | 🟢 Low |
| Concurrent access | How it's handled | function_name:line | 🔴 High |
Deep-Dive Highlights
Core mechanisms from this sub-module have been extracted into dedicated focus pages for in-depth exploration. Each page covers one algorithm, protocol, or subsystem with diagrams, code walks, and configuration details.
Deep-Dive Pages
Query Processing
sequenceDiagram
participant User as User Query
participant Tokenizer as DART Tokenizer
participant Analyzer as Query Analyzer
User->>Tokenizer: Raw text
Tokenizer->>Tokenizer: CJK-aware segments
Tokenizer->>Analyzer: Token list
Analyzer->>Analyzer: Weight terms
Analyzer->>User: Query representation
Deep-Dive Highlights
DART Trie Tokenizer Focus
Comprehensive deep-dive into the mechanisms and implementation details.
Code Examples
Example: Typical Usage
// Replace with actual code example
void example() {
// ...
}