Vision Module — Neural Layout & OCR Recognition

Deep learning models for document layout analysis, table structure recognition, and optical character recognition.

Core Recognizers (ONNX-Based Inference)

Architecture: Multi-Stage Recognition

Key Algorithms

Layout Recognition (DLA)

Model: Deep Layer Aggregation (DLA) backbone, trained on document images

• Input: Image (auto-scaled to 640×640)
• Processing: ResNet50 + FPN (Feature Pyramid Network) for multi-scale features
• Output: Bboxes {x, y, w, h, class_id, confidence}
• Inference: ONNX Runtime (CPU or GPU)
• Batch: Default 16 images at once
• Post-processing: NMS (threshold IoU=0.5), confidence filtering (thr=0.2)

OCR: Two-Stage

Stage 1: CRAFT Text Detection

• Detects word-level regions from image
• Outputs: list of {bbox, confidence}

Stage 2: PaddleOCR Recognition

• Per-region character recognition
• Language model inference
• Outputs: {text, confidence, bbox}
• Supports 80+ languages

Table Structure Recognition

Algorithm:

1. Apply Hough transform to detect lines (horizontal + vertical)
2. Extract line intersections as potential cell corners
3. Cluster intersections into grid
4. Infer cell merging (if adjacent cells have same label)
5. Output: {cell_bbox, row_idx, col_idx, row_span, col_span}

Integration Flow

From pdf_parser.py perspective:

1. Render page image (3x zoom)
2. Run LayoutRecognizer → get bboxes with class labels
3. Assign extracted text to bboxes (spatial overlap)
4. For each bbox by class:
   • Text: use assigned text or run OCR
   • Table: crop bbox region, run TSR, output grid
   • Figure: crop, save image, extract caption
   • Header/Footer: mark and optionally skip
5. OCR fallback for low-confidence text
6. Assemble into semantic blocks

Performance & Trade-offs

Configuration & Tuning

Environment Variables:

• LAYOUT_RECOGNIZER_TYPE — "onnx" (default) or "ascend"
• DEEPDOC_URL — Remote DLA server URL (fallback)
• TENSORRT_DLA_SVR — Alternative DLA server endpoint
• PARALLEL_DEVICES — Number of parallel inference workers
• HF_ENDPOINT — HuggingFace mirror (for model downloads)

Tunable Thresholds:

• layout_thr — Layout confidence threshold (default 0.2)
• ocr_thr — OCR confidence threshold (default 0.5)
• nms_iou — NMS IoU threshold (default 0.5)
• batch_size — Inference batch size (default 16)

Model Management

Local Model Loading

Models stored in:

RAGFlow_base_dir/rag/res/deepdoc/
├── layout.onnx (DLA model)
├── ...

HuggingFace Fallback

If local model missing, auto-download from:

https://huggingface.co/InfiniFlow/deepdoc

Remote DLA Server

Optional remote TensorRT inference via DEEPDOC_URL or TENSORRT_DLA_SVR environment variable.

Error Handling

• Model load fails (local) → Download from HuggingFace
• HuggingFace unavailable → Try remote DLA server
• Remote DLA unavailable → Fall back to empty layout (output raw text)
• OCR low confidence → Include text with confidence flag
• Table detection fails → Keep as unstructured text

Summary

The vision layer combines PaddlePaddle-trained DLA (layout), CRAFT (text detection), and PaddleOCR (text recognition) into a robust multi-stage pipeline. ONNX inference ensures portability and performance. Fallback strategies and configurable thresholds make the system resilient to edge cases—scanned PDFs, low-quality images, mixed layouts, and multilingual documents.