Oclgrind — Virtual OpenCL Device Simulator

LLVM-Based OpenCL Interpreter, Debugger & Analysis Framework

Oclgrind is a virtual OpenCL device simulator that interprets and executes OpenCL kernels on a software-emulated device built on LLVM IR. It provides a comprehensive platform for detecting memory access errors, data races, barrier divergence, and uninitialized memory usage in OpenCL programs, with an interactive kernel debugger and instruction profiling capabilities.

4
Source Modules
57
Source Files
~34k
Lines of Code
C++17
Language

LLVM 18+ IR Interpreter OpenCL 1.2 ICD Runtime 6 Analysis Plugins BSD 3-Clause License

📚 What is this & why does it exist?

Oclgrind is a software-based OpenCL device that simulates kernel execution by interpreting LLVM IR instructions one by one. Instead of running your OpenCL code on real GPU hardware, Oclgrind executes it in a controlled virtual environment where every memory access, every barrier, and every atomic operation can be inspected and validated. This enables detection of bugs that are nearly impossible to find on actual hardware.

💡 Real-world analogy

Think of Oclgrind like a slow-motion instant replay system for a sports match. Real GPU execution is like watching the game at full speed — you see the final score but miss the fouls. Oclgrind replays every single play frame-by-frame, with multiple camera angles (plugins) watching for rule violations: a referee for memory out-of-bounds (MemCheck), another for player collisions in the same lane (RaceDetector), and a coach's assistant tracking every move (InstructionCounter). It is slower than the live game, but it catches everything.

System Architecture

flowchart TD
  subgraph Runtime["Runtime Layer (ICD)"]
    RT["runtime.cpp
OpenCL API impl"] AQ["AsyncQueue
Command scheduling"] ICD["icd.h
ICD dispatch table"] end subgraph Core["Core Simulation Engine"] CTX["Context
Plugin manager"] PRG["Program
LLVM compilation"] KRN["Kernel
Argument binding"] KI["KernelInvocation
NDRange orchestrator"] WG["WorkGroup
Barrier sync"] WI["WorkItem
IR interpreter"] MEM["Memory
Address space mgr"] end subgraph Plugins["Analysis Plugins"] MC["MemCheck"] RD["RaceDetector"] UI["Uninitialized"] IC["InstructionCounter"] ID["InteractiveDebugger"] LG["Logger"] end subgraph KernelCLI["Kernel CLI"] SIM["Simulation
Standalone runner"] end RT -->|"clEnqueueNDRange"| AQ AQ -->|"dispatch"| CTX CTX -->|"compile"| PRG CTX -->|"invoke"| KI KI -->|"spawn"| WG WG -->|"execute"| WI WI -->|"load/store"| MEM CTX -->|"notify hooks"| MC CTX -->|"notify hooks"| RD CTX -->|"notify hooks"| UI CTX -->|"notify hooks"| IC CTX -->|"notify hooks"| ID CTX -->|"notify hooks"| LG SIM -->|"standalone exec"| CTX ICD -->|"dispatch"| RT

Kernel Execution Data Flow

sequenceDiagram
  participant App as Host Application
  participant RT as Runtime (ICD)
  participant CTX as Context
  participant PRG as Program
  participant KI as KernelInvocation
  participant WG as WorkGroup
  participant WI as WorkItem
  participant MEM as Memory
  participant PLG as Plugins

  App->>RT: clCreateProgramWithSource()
  RT->>CTX: Create Program
  CTX->>PRG: Compile OpenCL to LLVM IR
  PRG-->>CTX: Module ready
  App->>RT: clEnqueueNDRangeKernel()
  RT->>CTX: Queue kernel dispatch
  CTX->>PLG: notifyKernelBegin()
  CTX->>KI: Create invocation (global/local sizes)
  KI->>WG: Spawn work-groups (parallel or sequential)
  WG->>WI: Create work-items per group
  loop For each instruction
    WI->>WI: Interpret LLVM IR instruction
    WI->>MEM: Load/Store memory
    MEM-->>WI: Data returned
    WI->>CTX: notifyInstructionExecuted()
    CTX->>PLG: Forward to all plugins
  end
  WG->>CTX: notifyWorkGroupComplete()
  KI-->>CTX: All groups done
  CTX->>PLG: notifyKernelEnd()
  CTX-->>RT: Execution complete
  RT-->>App: Event signaled

🚶 Step-by-Step Narrated Walkthrough

Follow a single OpenCL vector addition kernel execution as it flows through Oclgrind:

  1. Host API Call — The application calls clCreateProgramWithSource(). The Runtime layer receives the OpenCL C source string and forwards it to the Core.
  2. Program Compilation — Program::build() invokes Clang to compile OpenCL C to LLVM IR. Precompiled headers for builtins are loaded from OCLGRIND_PCH_DIR. The resulting LLVM Module is stored.
  3. Kernel Creation — clCreateKernel("vecadd") extracts the named function from the Module. The Kernel object parses argument metadata (types, address spaces, sizes).
  4. Argument Binding — clSetKernelArg() binds each buffer and scalar. The Kernel stores typed argument values that will be loaded into private memory at launch.
  5. Enqueue & Dispatch — clEnqueueNDRangeKernel() pushes a command onto the AsyncQueue. The queue thread creates a KernelInvocation with global size, local size, and offset.
  6. Work-Group Spawning — KernelInvocation iterates over the NDRange grid. For each work-group, it allocates local Memory and creates a WorkGroup object containing N work-items.
  7. Work-Item Execution — Each WorkItem gets its own private Memory and begins interpreting LLVM IR instructions: load, store, add, getelementptr, etc. Every instruction triggers plugin callbacks.
  8. Memory Access Validation — On each load/store, MemCheck validates bounds, read/write permissions, and alignment. RaceDetector records the access in its shadow memory for conflict detection.
  9. Barrier Synchronization — When a work-item hits barrier(), the WorkGroup suspends it and switches to the next. Once all items reach the barrier, execution resumes. RaceDetector checks for divergent barriers.
  10. Plugin Analysis — InstructionCounter tallies each opcode. Uninitialized propagates shadow bits to detect reads of unwritten memory. InteractiveDebugger may pause at breakpoints.
  11. Work-Group Completion — After all work-items finish, WorkGroup deallocates local memory and notifies Context. Plugins receive workGroupComplete() — RaceDetector clears per-group shadow state.
  12. Kernel Completion — After all work-groups finish, KernelInvocation signals done. InstructionCounter prints histograms. The Runtime marks the event complete and the host can read results.

Key Components

ComponentFileRole
Contextsrc/core/Context.hCentral hub: owns global memory, manages plugin lifecycle, dispatches notification callbacks
Programsrc/core/Program.hCompiles OpenCL C to LLVM IR using Clang, manages binary caching and kernel extraction
Kernelsrc/core/Kernel.hRepresents a single kernel function with typed argument bindings and metadata
KernelInvocationsrc/core/KernelInvocation.hOrchestrates NDRange execution: spawns work-groups, manages multi-threading
WorkGroupsrc/core/WorkGroup.hContains work-items for one group, handles barrier synchronization and local memory
WorkItemsrc/core/WorkItem.hLLVM IR interpreter: executes instructions one-by-one for a single work-item
Memorysrc/core/Memory.hBuffer-based address space manager: allocate, load, store, atomic ops, bounds tracking
Pluginsrc/core/Plugin.hAbstract base class with virtual hooks for memory, instruction, and lifecycle events
MemChecksrc/plugins/MemCheck.hValidates memory access bounds, permissions, and alignment
RaceDetectorsrc/plugins/RaceDetector.hDetects data races between work-items using shadow memory tracking
Uninitializedsrc/plugins/Uninitialized.hTracks shadow bits to detect reads of uninitialized memory
InteractiveDebuggersrc/plugins/InteractiveDebugger.hGDB-like interactive kernel debugger with breakpoints, stepping, memory inspection
InstructionCountersrc/plugins/InstructionCounter.hCollects per-opcode execution histograms across all work-items
Simulationsrc/kernel/Simulation.hStandalone kernel runner: loads program, sets args, executes without full runtime
AsyncQueuesrc/runtime/async_queue.hThread-safe command queue for asynchronous kernel dispatch and memory operations

Cross-Module Connections

sequenceDiagram
  participant RT as Runtime (ICD)
  participant CTX as Core::Context
  participant MEM as Core::Memory
  participant PLG as Plugins
  participant KRN as Kernel CLI

  RT->>CTX: clEnqueueNDRangeKernel
  Note over RT,CTX: Runtime translates OpenCL API to Core calls
  CTX->>MEM: allocateBuffer / load / store
  Note over CTX,MEM: Memory validates all accesses
  CTX->>PLG: notifyMemoryLoad / notifyMemoryStore
  Note over CTX,PLG: Every memory op forwarded to active plugins
  PLG-->>CTX: logError (if violation detected)
  KRN->>CTX: Simulation::run (standalone path)
  Note over KRN,CTX: CLI bypasses Runtime, talks directly to Core
Key insight: The Core module is the center of everything — it owns the simulation state and dispatches events. The Runtime translates OpenCL API calls into Core operations. Plugins are passive observers that receive callbacks from Core. The Kernel CLI provides a standalone entry point that bypasses the Runtime entirely.

Modules

Core Simulation

core/

The simulation engine: LLVM IR interpreter, memory model, work-item/group execution, and plugin notification system.

  • src/core/ · 21 files · ~12k lines
  • Context, Memory, WorkItem, Program, Kernel
  • OpenCL address space emulation

Analysis & Debugging

plugins/

Six analysis plugins: memory checking, data race detection, uninitialized access tracking, instruction counting, interactive debugging, and logging.

  • src/plugins/ · 12 files · ~5.6k lines
  • MemCheck, RaceDetector, Uninitialized
  • Shadow memory & happens-before analysis

Host Interface

runtime/

OpenCL ICD-compliant runtime: implements all cl* API functions, asynchronous command queue, and device/platform queries.

  • src/runtime/ · 5 files · ~7.4k lines
  • Full OpenCL 1.2 API surface
  • ICD dispatch table & async execution

kernel/

Standalone kernel execution CLI (oclgrind-kernel): command-line tool for running kernels outside an OpenCL host program.

  • src/kernel/ · 3 files · ~1.2k lines
  • Simulation class for direct execution
  • Command-line argument parsing

Configuration Overview

Environment VariableDefaultDescription
OCLGRIND_INST_COUNTSoffEnable instruction counting plugin
OCLGRIND_DATA_RACESoffEnable data race detection plugin
OCLGRIND_UNINITIALIZEDoffEnable uninitialized memory detection
OCLGRIND_INTERACTIVEoffEnable interactive debugger plugin
OCLGRIND_PLUGINS—Comma-separated list of dynamic plugin shared libraries to load
OCLGRIND_NUM_THREADSHW coresNumber of threads for parallel work-group execution
OCLGRIND_QUICKoffExecute only one work-group (fast smoke test)
OCLGRIND_BUILD_OPTIONS—Extra Clang compiler options appended to kernel builds
OCLGRIND_LOGstderrFile path for Logger plugin output
OCLGRIND_MAX_ERRORS1000Maximum errors reported before suppression
OCLGRIND_PCH_DIRinstall/Directory containing precompiled OpenCL builtin headers
OCLGRIND_DUMP_SPIRoffDump compiled SPIR-V / LLVM IR to temporary files
OCLGRIND_DISABLE_PCHoffDisable precompiled headers (slower builds)
OCLGRIND_GLOBAL_MEM_SIZE1 GBSimulated global memory size
OCLGRIND_LOCAL_MEM_SIZE32 KBSimulated local memory size per work-group

Directory Map

Source tree for Oclgrind
Oclgrind/
├── src/
│   ├── core/                      ← Simulation engine (Context, Memory, WorkItem, Program)
│   │   ├── Context.h / .cpp       ← Central hub, plugin dispatch
│   │   ├── Memory.h / .cpp        ← Buffer-based address space manager
│   │   ├── WorkItem.h / .cpp      ← LLVM IR instruction interpreter
│   │   ├── WorkItemBuiltins.cpp   ← OpenCL builtin function implementations
│   │   ├── WorkGroup.h / .cpp     ← Barrier sync, local memory
│   │   ├── KernelInvocation.h/.cpp ← NDRange orchestration
│   │   ├── Kernel.h / .cpp        ← Kernel argument binding
│   │   ├── Program.h / .cpp       ← Clang compilation pipeline
│   │   ├── Plugin.h / .cpp        ← Abstract plugin base class
│   │   ├── Queue.h / .cpp         ← Command queue
│   │   └── common.h / .cpp        ← Shared types, enums, utilities
│   ├── plugins/                   ← Analysis plugins
│   │   ├── MemCheck.h / .cpp      ← Memory bounds & permission checking
│   │   ├── RaceDetector.h / .cpp  ← Data race detection (shadow memory)
│   │   ├── Uninitialized.h / .cpp ← Uninitialized memory tracking (shadow bits)
│   │   ├── InstructionCounter.h/.cpp ← Opcode histogram profiler
│   │   ├── InteractiveDebugger.h/.cpp ← GDB-like kernel debugger
│   │   └── Logger.h / .cpp        ← Message logging to file
│   ├── runtime/                   ← OpenCL ICD runtime
│   │   ├── runtime.cpp            ← Full cl* API implementation (~5.6k lines)
│   │   ├── async_queue.h / .cpp   ← Thread-safe async command queue
│   │   ├── icd.h                  ← ICD dispatch table definition
│   │   └── oclgrind.cpp           ← oclgrind launcher (LD_PRELOAD injection)
│   ├── kernel/                    ← Standalone CLI
│   │   ├── Simulation.h / .cpp    ← Direct kernel execution engine
│   │   └── oclgrind-kernel.cpp    ← CLI argument parsing & entry point
│   └── CL/                        ← OpenCL API headers (cl.h, cl_ext.h, etc.)
├── tests/                         ← Test suite (kernels + expected results)
├── CMakeLists.txt                 ← CMake build system (LLVM/Clang dependency)
└── README.md                      ← Project documentation
LevelWhat You FindWhen to Use
This page (L1)Project overview, architecture, module cardsStart here to understand the project structure
Module pages (L2)Deep-dive into each module: classes, data flow, algorithms, configurationWhen you need to understand a specific module
Focus pagesAlgorithm deep-dives: memory model, race detection, IR interpretationWhen you need implementation-level detail on a mechanism
SearchFull-text search across all wiki pagesWhen you know what you are looking for