C++ Text Analysis: High-Performance Tokenization

Core Components

Architecture: Go ↔ C++ Bridge

Go Handler/Service
    |
    v
Tokenizer wrapper (tokenizer.go)
    | cgo boundary
    v
C Shim API (rag_analyzer_c_api.h)
    | FFI
    v
C++ Implementation
 ├─ darts_trie.cpp
 ├─ tokenizer.cpp
 ├─ lemmatizer.cpp
 └─ opencc_wrapper.cpp
    |
    v
Return marshaled results to Go

DARTS Double-Array Trie

What is DARTS?

Double-Array Trie Strings (DARTS) is a memory-efficient data structure for dictionary lookup. Instead of tree nodes, it uses two integer arrays (base and check) that fit in CPU cache.

Why DARTS over HashMap?

DARTS Lookup Algorithm

Given string "hello" and DARTS state machine:

state = 0
for each character c in string:
    idx = base[state] + (c value)
    if check[idx] == state:
        state = idx
    else:
        return NOT_FOUND

if final_state is marked END:
    return FOUND
else:
    return NOT_FOUND

Time complexity: O(string_length * ~20ns), not O(string_length^2) like hashmap worst-case.

Tokenization Pipeline

Input: Raw Text

"今天天气很好，我想去公园。"
(Today the weather is very nice, I want to go to the park.)

Step 1: Unicode Normalization

Decompose characters, remove accents, normalize spaces.

Step 2: DARTS Lookup

For each substring, check DARTS dictionary. Identify longest matching word at each position (greedy longest-match).

Step 3: Part-of-Speech Tagging

Each matched token is assigned a part-of-speech (noun, verb, adjective, etc.)

Step 4: English Lemmatization

If token contains Latin characters, apply lemmatization rules.

Output: Token List

[
  { "token": "今天", "pos": "NOUN", "score": 1.0 },
  { "token": "天气", "pos": "NOUN", "score": 1.0 },
  { "token": "很", "pos": "ADV", "score": 0.8 },
  { "token": "好", "pos": "ADJ", "score": 1.0 },
]

cgo Integration

C API Definition

// rag_analyzer_c_api.h
extern "C" {
    struct TokenResult {
        char* token;
        char* pos;
        float score;
    };
    
    struct TokenizeResult {
        TokenResult* tokens;
        int count;
    };
    
    TokenizeResult tokenize(const char* text, int len);
    void free_result(TokenizeResult result);
}

Go Bindings

// tokenizer.go
package binding

import "C"

func Tokenize(text string) ([]Token, error) {
    cText := C.CString(text)
    defer C.free(unsafe.Pointer(cText))
    
    result := C.tokenize(cText, C.int(len(text)))
    defer C.free_result(result)
    
    tokens := make([]Token, result.count)
    for i := 0; i < int(result.count); i++ {
        tokens[i] = Token{
            Word: C.GoString(result.tokens[i].token),
            POS: C.GoString(result.tokens[i].pos),
            Score: float32(result.tokens[i].score),
        }
    }
    return tokens, nil
}

Files Overview

Performance Tuning

Memory Layout

DARTS arrays are mmap'ed from disk, keeping heap pressure low. CPP tokenizer avoids dynamic allocations in the hot path.

Batch Processing

Call tokenize() once per batch (multiple sentences) rather than once per sentence to amortize cgo overhead.

Caching Tokenization Results

Service layer caches tokenized results per query string (embedding lookup cache).

Building & Dependencies

The C++ code depends on:

• DARTS library: For trie construction and lookup
• OpenCC library: Chinese conversion
• RE2 library: Safe regex
• Boost Unicode: UTF-8 handling

Build via CMake:

cd internal/cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
# Outputs: librag_analyzer.a, librag_analyzer.so

Integration Checklist

• cgo CFLAGS point to C++ headers
• cgo LDFLAGS link librag_analyzer.a (or .so)
• Go bindings use unsafe.Pointer, defer C.free()
• C function signatures match exactly
• Malloc/free in C, not in Go (ownership)

cgo Best Practices

Memory Management

C++ allocates, Go frees. Never let Go garbage collect C-allocated memory:

// WRONG: Go pointer to C memory, may crash if GC moves Go object
var cStr *C.char = C.malloc(100)
goStr := C.GoString(cStr)  // dangerous!

// CORRECT: Store C pointer, free explicitly
cStr := C.malloc(100)
defer C.free(unsafe.Pointer(cStr))
goStr := C.GoString(cStr)

Build Configuration

Compile C++ with optimization flags:

// In go file
// #cgo CFLAGS: -I/usr/include/rag_analyzer -O3
// #cgo LDFLAGS: -L/usr/lib -lrag_analyzer
// #include "rag_analyzer.h"
import "C"

go build  // automatically applies CFLAGS/LDFLAGS

Error Handling Across Boundary

C++ exceptions don't cross the cgo boundary. Use return codes:

// C API
typedef struct {
    int code;  // 0 = success, 1 = error
    char* message;
} Result;

// Go wrapper
result := C.tokenize(cText)
defer C.free(unsafe.Pointer(result.message))

if result.code != 0 {
    return nil, errors.New(C.GoString(result.message))
}

Integration Patterns

Lazy Initialization

Load DARTS dictionary once at startup:

var (
    dartsOnce sync.Once
    dartsTrie *Trie
)

func GetDARTS() *Trie {
    dartsOnce.Do(func() {
        dartsTrie = C.load_darts("/path/to/darts.bin")
    })
    return dartsTrie
}

Timeout for C++ Calls

Long-running C++ operations can block Go scheduler. Use goroutines + channels:

func TokenizeWithTimeout(ctx context.Context, text string, timeout time.Duration) ([]Token, error) {
    result := make(chan []Token, 1)
    errChan := make(chan error, 1)
    
    go func() {
        tokens, err := Tokenize(text)  // may be slow
        if err != nil {
            errChan <- err
        } else {
            result <- tokens
        }
    }()
    
    select {
    case <-ctx.Done():
        return nil, ctx.Err()
    case tokens := <-result:
        return tokens, nil
    case err := <-errChan:
        return nil, err
    case <-time.After(timeout):
        return nil, errors.New("tokenization timeout")
    }
}

Further Reading

• DARTS Deep Dive — Algorithm walkthrough
• Internal Module Overview