Oclgrind — Virtual OpenCL Device Simulator
LLVM-Based OpenCL Interpreter, Debugger & Analysis Framework
Oclgrind is a virtual OpenCL device simulator that interprets and executes OpenCL kernels on a software-emulated device built on LLVM IR. It provides a comprehensive platform for detecting memory access errors, data races, barrier divergence, and uninitialized memory usage in OpenCL programs, with an interactive kernel debugger and instruction profiling capabilities.
LLVM 18+ IR Interpreter OpenCL 1.2 ICD Runtime 6 Analysis Plugins BSD 3-Clause License
📚 What is this & why does it exist?
Oclgrind is a software-based OpenCL device that simulates kernel execution by interpreting LLVM IR instructions one by one. Instead of running your OpenCL code on real GPU hardware, Oclgrind executes it in a controlled virtual environment where every memory access, every barrier, and every atomic operation can be inspected and validated. This enables detection of bugs that are nearly impossible to find on actual hardware.
💡 Real-world analogy
Think of Oclgrind like a slow-motion instant replay system for a sports match. Real GPU execution is like watching the game at full speed — you see the final score but miss the fouls. Oclgrind replays every single play frame-by-frame, with multiple camera angles (plugins) watching for rule violations: a referee for memory out-of-bounds (MemCheck), another for player collisions in the same lane (RaceDetector), and a coach's assistant tracking every move (InstructionCounter). It is slower than the live game, but it catches everything.
System Architecture
flowchart TD
subgraph Runtime["Runtime Layer (ICD)"]
RT["runtime.cpp
OpenCL API impl"]
AQ["AsyncQueue
Command scheduling"]
ICD["icd.h
ICD dispatch table"]
end
subgraph Core["Core Simulation Engine"]
CTX["Context
Plugin manager"]
PRG["Program
LLVM compilation"]
KRN["Kernel
Argument binding"]
KI["KernelInvocation
NDRange orchestrator"]
WG["WorkGroup
Barrier sync"]
WI["WorkItem
IR interpreter"]
MEM["Memory
Address space mgr"]
end
subgraph Plugins["Analysis Plugins"]
MC["MemCheck"]
RD["RaceDetector"]
UI["Uninitialized"]
IC["InstructionCounter"]
ID["InteractiveDebugger"]
LG["Logger"]
end
subgraph KernelCLI["Kernel CLI"]
SIM["Simulation
Standalone runner"]
end
RT -->|"clEnqueueNDRange"| AQ
AQ -->|"dispatch"| CTX
CTX -->|"compile"| PRG
CTX -->|"invoke"| KI
KI -->|"spawn"| WG
WG -->|"execute"| WI
WI -->|"load/store"| MEM
CTX -->|"notify hooks"| MC
CTX -->|"notify hooks"| RD
CTX -->|"notify hooks"| UI
CTX -->|"notify hooks"| IC
CTX -->|"notify hooks"| ID
CTX -->|"notify hooks"| LG
SIM -->|"standalone exec"| CTX
ICD -->|"dispatch"| RT
Kernel Execution Data Flow
sequenceDiagram
participant App as Host Application
participant RT as Runtime (ICD)
participant CTX as Context
participant PRG as Program
participant KI as KernelInvocation
participant WG as WorkGroup
participant WI as WorkItem
participant MEM as Memory
participant PLG as Plugins
App->>RT: clCreateProgramWithSource()
RT->>CTX: Create Program
CTX->>PRG: Compile OpenCL to LLVM IR
PRG-->>CTX: Module ready
App->>RT: clEnqueueNDRangeKernel()
RT->>CTX: Queue kernel dispatch
CTX->>PLG: notifyKernelBegin()
CTX->>KI: Create invocation (global/local sizes)
KI->>WG: Spawn work-groups (parallel or sequential)
WG->>WI: Create work-items per group
loop For each instruction
WI->>WI: Interpret LLVM IR instruction
WI->>MEM: Load/Store memory
MEM-->>WI: Data returned
WI->>CTX: notifyInstructionExecuted()
CTX->>PLG: Forward to all plugins
end
WG->>CTX: notifyWorkGroupComplete()
KI-->>CTX: All groups done
CTX->>PLG: notifyKernelEnd()
CTX-->>RT: Execution complete
RT-->>App: Event signaled
🚶 Step-by-Step Narrated Walkthrough
Follow a single OpenCL vector addition kernel execution as it flows through Oclgrind:
- Host API Call — The application calls
clCreateProgramWithSource(). The Runtime layer receives the OpenCL C source string and forwards it to the Core. - Program Compilation —
Program::build()invokes Clang to compile OpenCL C to LLVM IR. Precompiled headers for builtins are loaded fromOCLGRIND_PCH_DIR. The resulting LLVM Module is stored. - Kernel Creation —
clCreateKernel("vecadd")extracts the named function from the Module. TheKernelobject parses argument metadata (types, address spaces, sizes). - Argument Binding —
clSetKernelArg()binds each buffer and scalar. The Kernel stores typed argument values that will be loaded into private memory at launch. - Enqueue & Dispatch —
clEnqueueNDRangeKernel()pushes a command onto theAsyncQueue. The queue thread creates aKernelInvocationwith global size, local size, and offset. - Work-Group Spawning —
KernelInvocationiterates over the NDRange grid. For each work-group, it allocates localMemoryand creates aWorkGroupobject containing N work-items. - Work-Item Execution — Each
WorkItemgets its own privateMemoryand begins interpreting LLVM IR instructions:load,store,add,getelementptr, etc. Every instruction triggers plugin callbacks. - Memory Access Validation — On each
load/store,MemCheckvalidates bounds, read/write permissions, and alignment.RaceDetectorrecords the access in its shadow memory for conflict detection. - Barrier Synchronization — When a work-item hits
barrier(), theWorkGroupsuspends it and switches to the next. Once all items reach the barrier, execution resumes.RaceDetectorchecks for divergent barriers. - Plugin Analysis —
InstructionCountertallies each opcode.Uninitializedpropagates shadow bits to detect reads of unwritten memory.InteractiveDebuggermay pause at breakpoints. - Work-Group Completion — After all work-items finish,
WorkGroupdeallocates local memory and notifies Context. Plugins receiveworkGroupComplete()—RaceDetectorclears per-group shadow state. - Kernel Completion — After all work-groups finish,
KernelInvocationsignals done.InstructionCounterprints histograms. The Runtime marks the event complete and the host can read results.
Key Components
| Component | File | Role |
|---|---|---|
Context | src/core/Context.h | Central hub: owns global memory, manages plugin lifecycle, dispatches notification callbacks |
Program | src/core/Program.h | Compiles OpenCL C to LLVM IR using Clang, manages binary caching and kernel extraction |
Kernel | src/core/Kernel.h | Represents a single kernel function with typed argument bindings and metadata |
KernelInvocation | src/core/KernelInvocation.h | Orchestrates NDRange execution: spawns work-groups, manages multi-threading |
WorkGroup | src/core/WorkGroup.h | Contains work-items for one group, handles barrier synchronization and local memory |
WorkItem | src/core/WorkItem.h | LLVM IR interpreter: executes instructions one-by-one for a single work-item |
Memory | src/core/Memory.h | Buffer-based address space manager: allocate, load, store, atomic ops, bounds tracking |
Plugin | src/core/Plugin.h | Abstract base class with virtual hooks for memory, instruction, and lifecycle events |
MemCheck | src/plugins/MemCheck.h | Validates memory access bounds, permissions, and alignment |
RaceDetector | src/plugins/RaceDetector.h | Detects data races between work-items using shadow memory tracking |
Uninitialized | src/plugins/Uninitialized.h | Tracks shadow bits to detect reads of uninitialized memory |
InteractiveDebugger | src/plugins/InteractiveDebugger.h | GDB-like interactive kernel debugger with breakpoints, stepping, memory inspection |
InstructionCounter | src/plugins/InstructionCounter.h | Collects per-opcode execution histograms across all work-items |
Simulation | src/kernel/Simulation.h | Standalone kernel runner: loads program, sets args, executes without full runtime |
AsyncQueue | src/runtime/async_queue.h | Thread-safe command queue for asynchronous kernel dispatch and memory operations |
Cross-Module Connections
sequenceDiagram participant RT as Runtime (ICD) participant CTX as Core::Context participant MEM as Core::Memory participant PLG as Plugins participant KRN as Kernel CLI RT->>CTX: clEnqueueNDRangeKernel Note over RT,CTX: Runtime translates OpenCL API to Core calls CTX->>MEM: allocateBuffer / load / store Note over CTX,MEM: Memory validates all accesses CTX->>PLG: notifyMemoryLoad / notifyMemoryStore Note over CTX,PLG: Every memory op forwarded to active plugins PLG-->>CTX: logError (if violation detected) KRN->>CTX: Simulation::run (standalone path) Note over KRN,CTX: CLI bypasses Runtime, talks directly to Core
Key insight: The Core module is the center of everything — it owns the simulation state and dispatches events. The Runtime translates OpenCL API calls into Core operations. Plugins are passive observers that receive callbacks from Core. The Kernel CLI provides a standalone entry point that bypasses the Runtime entirely.
Modules
Core Simulation
Analysis & Debugging
Host Interface
runtime/
OpenCL ICD-compliant runtime: implements all cl* API functions, asynchronous command queue, and device/platform queries.
- src/runtime/ · 5 files · ~7.4k lines
- Full OpenCL 1.2 API surface
- ICD dispatch table & async execution
kernel/
Standalone kernel execution CLI (oclgrind-kernel): command-line tool for running kernels outside an OpenCL host program.
- src/kernel/ · 3 files · ~1.2k lines
- Simulation class for direct execution
- Command-line argument parsing
Configuration Overview
| Environment Variable | Default | Description |
|---|---|---|
OCLGRIND_INST_COUNTS | off | Enable instruction counting plugin |
OCLGRIND_DATA_RACES | off | Enable data race detection plugin |
OCLGRIND_UNINITIALIZED | off | Enable uninitialized memory detection |
OCLGRIND_INTERACTIVE | off | Enable interactive debugger plugin |
OCLGRIND_PLUGINS | — | Comma-separated list of dynamic plugin shared libraries to load |
OCLGRIND_NUM_THREADS | HW cores | Number of threads for parallel work-group execution |
OCLGRIND_QUICK | off | Execute only one work-group (fast smoke test) |
OCLGRIND_BUILD_OPTIONS | — | Extra Clang compiler options appended to kernel builds |
OCLGRIND_LOG | stderr | File path for Logger plugin output |
OCLGRIND_MAX_ERRORS | 1000 | Maximum errors reported before suppression |
OCLGRIND_PCH_DIR | install/ | Directory containing precompiled OpenCL builtin headers |
OCLGRIND_DUMP_SPIR | off | Dump compiled SPIR-V / LLVM IR to temporary files |
OCLGRIND_DISABLE_PCH | off | Disable precompiled headers (slower builds) |
OCLGRIND_GLOBAL_MEM_SIZE | 1 GB | Simulated global memory size |
OCLGRIND_LOCAL_MEM_SIZE | 32 KB | Simulated local memory size per work-group |
Directory Map
Source tree for Oclgrind
Oclgrind/
├── src/
│ ├── core/ ← Simulation engine (Context, Memory, WorkItem, Program)
│ │ ├── Context.h / .cpp ← Central hub, plugin dispatch
│ │ ├── Memory.h / .cpp ← Buffer-based address space manager
│ │ ├── WorkItem.h / .cpp ← LLVM IR instruction interpreter
│ │ ├── WorkItemBuiltins.cpp ← OpenCL builtin function implementations
│ │ ├── WorkGroup.h / .cpp ← Barrier sync, local memory
│ │ ├── KernelInvocation.h/.cpp ← NDRange orchestration
│ │ ├── Kernel.h / .cpp ← Kernel argument binding
│ │ ├── Program.h / .cpp ← Clang compilation pipeline
│ │ ├── Plugin.h / .cpp ← Abstract plugin base class
│ │ ├── Queue.h / .cpp ← Command queue
│ │ └── common.h / .cpp ← Shared types, enums, utilities
│ ├── plugins/ ← Analysis plugins
│ │ ├── MemCheck.h / .cpp ← Memory bounds & permission checking
│ │ ├── RaceDetector.h / .cpp ← Data race detection (shadow memory)
│ │ ├── Uninitialized.h / .cpp ← Uninitialized memory tracking (shadow bits)
│ │ ├── InstructionCounter.h/.cpp ← Opcode histogram profiler
│ │ ├── InteractiveDebugger.h/.cpp ← GDB-like kernel debugger
│ │ └── Logger.h / .cpp ← Message logging to file
│ ├── runtime/ ← OpenCL ICD runtime
│ │ ├── runtime.cpp ← Full cl* API implementation (~5.6k lines)
│ │ ├── async_queue.h / .cpp ← Thread-safe async command queue
│ │ ├── icd.h ← ICD dispatch table definition
│ │ └── oclgrind.cpp ← oclgrind launcher (LD_PRELOAD injection)
│ ├── kernel/ ← Standalone CLI
│ │ ├── Simulation.h / .cpp ← Direct kernel execution engine
│ │ └── oclgrind-kernel.cpp ← CLI argument parsing & entry point
│ └── CL/ ← OpenCL API headers (cl.h, cl_ext.h, etc.)
├── tests/ ← Test suite (kernels + expected results)
├── CMakeLists.txt ← CMake build system (LLVM/Clang dependency)
└── README.md ← Project documentation
Navigation Guide
| Level | What You Find | When to Use |
|---|---|---|
| This page (L1) | Project overview, architecture, module cards | Start here to understand the project structure |
| Module pages (L2) | Deep-dive into each module: classes, data flow, algorithms, configuration | When you need to understand a specific module |
| Focus pages | Algorithm deep-dives: memory model, race detection, IR interpretation | When you need implementation-level detail on a mechanism |
| Search | Full-text search across all wiki pages | When you know what you are looking for |