GPGPU-Sim

Cycle-Level GPU Simulator for CUDA & OpenCL Workloads

GPGPU-Sim is a detailed, cycle-level simulator that models contemporary graphics processing units (GPUs) running compute workloads written in CUDA or OpenCL. It includes an interconnection network simulator (BookSim-based InterSim2), a configurable power model (AccelWattch/McPAT), and a performance visualization tool (AerialVision). GPGPU-Sim has been rigorously validated against real NVIDIA hardware from various generations.

🎯 Cycle-Level Accuracy 🔌 CUDA 4.2 – 11 ⚡ Accel-Sim Compatible 💡 AccelWattch Power Model

📚 What is this & why does it exist?

GPGPU-Sim is a research simulator that lets you run real CUDA and OpenCL programs on a virtual GPU, modeling every cycle of the hardware pipeline. It exists because studying GPU microarchitecture on real hardware is extremely difficult — the internals are proprietary and invisible. GPGPU-Sim makes the GPU transparent: every cache miss, every warp stall, every DRAM bank conflict is observable and configurable.

💡 Real-world analogy

Imagine you want to understand how a massive airport operates — from check-in to landing. You cannot observe the real airport’s internal systems directly, so you build a detailed scale model where every gate, runway, baggage conveyor, and control tower operates exactly as the real one does. You can now change the number of runways, add terminals, reroute baggage, and watch the exact second-by-second consequences. GPGPU-Sim is that scale model for GPUs: it replaces the opaque silicon chip with a transparent software replica.

Modules

131k

Lines of Code

C/C++

Language

300

Source Files

System Architecture

flowchart TD
  subgraph APP["Application Layer"]
    CUDA["CUDA Application"]
    OCL["OpenCL Application"]
  end
  subgraph RUNTIME["Runtime Libraries"]
    LIBCUDA["libcuda
CUDA API Interception"]
    LIBOCL["libopencl
OpenCL API"]
  end
  subgraph ENTRY["Simulator Entry"]
    EP["gpgpusim_entrypoint
Stream Manager"]
    AHM["abstract_hardware_model
Kernel, Warp, Instruction Abstractions"]
  end
  subgraph CORE["GPU Core Simulation (gpgpu-sim)"]
    GPUSIM["gpu-sim
Top-Level Simulator"]
    SHADER["shader
Shader Core Array & Warp Schedulers"]
    CACHE["gpu-cache
L1/Texture/Const Caches"]
    L2["l2cache
L2 Cache Partitions"]
    DRAM["dram / dram_sched
DRAM Timing & Scheduling"]
    SCORE["scoreboard
Warp Dependency Tracking"]
  end
  subgraph FUNC["Functional Simulation (cuda-sim)"]
    PTXLD["ptx_loader
PTX Binary Loading"]
    PTXIR["ptx_ir
IR Representation"]
    PTXSIM["ptx_sim
Warp Execution"]
    INSTR["instructions
PTX Opcode Execution"]
    MEM["memory
Address Space Model"]
  end
  subgraph ICNT["Interconnect (InterSim2)"]
    ICNTI["interconnect_interface"]
    TM["trafficmanager
Flit Injection & Routing"]
    NET["networks
Mesh, Torus, Fly, Fat-tree"]
    RTR["routers
IQ, Event, Chaos"]
  end
  subgraph POWER["Power Model (AccelWattch)"]
    WRAP["gpgpu_sim_wrapper
Power Interface"]
    PROC["processor / core
McPAT Components"]
    CACTI["cacti
Cache Area & Timing"]
  end
  CUDA --> LIBCUDA
  OCL --> LIBOCL
  LIBCUDA --> EP
  LIBOCL --> EP
  EP --> AHM
  AHM --> GPUSIM
  GPUSIM --> SHADER
  SHADER --> SCORE
  SHADER --> CACHE
  SHADER --> PTXSIM
  PTXSIM --> PTXIR
  PTXSIM --> INSTR
  INSTR --> MEM
  CACHE --> L2
  L2 --> ICNTI
  ICNTI --> TM
  TM --> NET
  TM --> RTR
  L2 --> DRAM
  GPUSIM --> WRAP
  WRAP --> PROC
  PROC --> CACTI

Core Simulation Engine

gpgpu-sim

GPU Timing Simulation — Shader Cores, Caches, DRAM

The heart of the simulator: models shader core arrays with warp schedulers, multi-level cache hierarchy (L1/L2), DRAM timing with bank-level parallelism, and the scoreboard for dependency tracking. Drives cycle-by-cycle execution of GPU kernels.

46 files25.5k LOCCaches · DRAM · Scheduler

cuda-sim

PTX Functional Simulation — Instruction Decode & Execute

Functionally simulates PTX instructions: loads PTX binaries, builds an IR, and executes each instruction including memory operations, control flow, and special functions. Provides the functional correctness layer that feeds the timing model.

28 files24.5k LOCPTX · Warp Execution

Core Infrastructure

Hardware Abstraction, Entry Point & Stream Management

The foundational layer: abstract_hardware_model defines kernel, warp, and instruction abstractions; gpgpusim_entrypoint bootstraps simulation; stream_manager handles CUDA streams and concurrency; option_parser manages configuration.

15 files5.3k LOCAbstractions · Config

Interconnect & Power

InterSim2

Interconnection Network Simulator (BookSim-based)

A full NoC simulator integrated into GPGPU-Sim: models various network topologies (mesh, torus, butterfly, fat-tree, dragonfly), virtual-channel routers, arbitration schemes, and credit-based flow control. Simulates flit-level traffic between shader cores and memory partitions.

115 files26.6k LOC7 topologies · 5 allocators

AccelWattch

GPU Power Modeling Framework (McPAT/CACTI-based)

Models GPU power consumption using analytical models derived from McPAT and CACTI. Estimates dynamic and leakage power for processor cores, shared caches, memory controllers, NoC, and I/O controllers. Validated against NVIDIA Volta hardware.

72 files49.4k LOCMcPAT · CACTI · Volta-validated

Runtime Libraries & Tools

libcuda

CUDA Runtime API Interception Library

Intercepts CUDA API calls from applications and redirects them to the GPGPU-Sim simulator instead of real GPU hardware. Handles kernel launches, memory allocation, and device management.

7 filesCUDA API

libopencl

OpenCL Runtime API Interception

Provides OpenCL API compatibility, allowing OpenCL workloads to run on the simulated GPU. Translates OpenCL constructs to the internal simulation model.

2 filesOpenCL API

cuobjdump_to_ptxplus

CUDA Binary Disassembly & PTXPlus Conversion

Parses cuobjdump output to extract and convert CUDA binary code into PTXPlus format, enabling simulation of pre-compiled CUDA binaries and SASS instructions.

14 filesLex/Yacc parsers

AerialVision

Performance Visualization Tool

A Python-based GUI tool for visualizing GPGPU-Sim simulation output: warp activity, memory access patterns, IPC over time, and other performance metrics. Uses log file parsing and interactive plotting.

9 filesPython · GUI

Level	What You’ll Find	When to Use
0 — Project Hub (this page)	Project overview, module map, architecture diagrams	Getting started, finding the right module
1 — Module Overview	Architecture diagram, key components, sub-module cards, narrated walkthrough	Understanding a module’s purpose and structure
2 — Sub-Module Deep Dive	Source-level classes, algorithms, configs, code examples, focus pages	Implementing, debugging, or extending code

How the Modules Connect — A Memory Request Journey

This diagram traces a single memory load instruction from the application layer through functional simulation, timing simulation, the interconnect, and DRAM — showing how all major modules collaborate.

sequenceDiagram
    participant APP as CUDA App
    participant LIB as libcuda
    participant SM as Stream Manager
    participant SH as Shader Core
(gpgpu-sim)
    participant FS as ptx_sim
(cuda-sim)
    participant L1 as L1 Data Cache
(gpu-cache)
    participant ICNT as InterSim2
(interconnect)
    participant L2 as L2 Cache
(l2cache)
    participant DR as DRAM
(dram)
    participant PW as AccelWattch
(power)

    APP->>LIB: cudaMemcpy / kernel launch
    LIB->>SM: enqueue kernel to stream
    SM->>SH: dispatch thread blocks to cores
    SH->>FS: functional execute instruction
    FS-->>SH: effective address computed
    SH->>L1: issue memory request (mem_fetch)
    Note over L1: L1 HIT: return data in ~28 cycles
    L1->>ICNT: L1 MISS: inject request flit
    ICNT->>L2: route flit to memory partition
    Note over ICNT: Mesh/crossbar traversal ~20-100 cycles
    L2->>DR: L2 MISS: issue DRAM command
    DR-->>L2: data returned after tRAS+tCAS
    L2-->>ICNT: reply flit with data
    ICNT-->>L1: deliver to requesting core
    L1-->>SH: unblock waiting warp
    SH->>PW: report activity counters
    PW-->>SH: power/energy estimate

Key insight: The boundary between functional and timing simulation is critical: cuda-sim computes what happens (correct addresses, values), while gpgpu-sim determines when it happens (cycle-accurate timing). The interconnect (InterSim2) faithfully models the latency and contention of moving data between cores and memory. AccelWattch observes hardware activity counters to estimate power.

Directory Map

Project source tree

gpgpu-sim_distribution/
├── src/
│   ├── abstract_hardware_model.h/cc   ← Role: Kernel, warp, instruction abstractions
│   ├── gpgpusim_entrypoint.h/cc       ← Role: Simulator bootstrap & main loop
│   ├── stream_manager.h/cc            ← Role: CUDA stream & event management
│   ├── option_parser.h/cc             ← Role: Configuration file parsing
│   ├── debug.h/cc                     ← Role: Debug output utilities
│   ├── trace.h/cc                     ← Role: Simulation trace infrastructure
│   ├── gpgpu-sim/                     ← Role: Core timing simulation (46 files)
│   ├── cuda-sim/                      ← Role: PTX functional simulation (28 files)
│   ├── intersim2/                     ← Role: Interconnect network simulator (115 files)
│   └── accelwattch/                   ← Role: Power modeling (72 files)
├── libcuda/                           ← Role: CUDA runtime API interception
├── libopencl/                         ← Role: OpenCL runtime API
├── cuobjdump_to_ptxplus/              ← Role: Binary disassembly tool
├── aerialvision/                      ← Role: Python visualization GUI
├── configs/                           ← Role: Pre-built GPU configuration files
├── doc/                               ← Role: Documentation
├── README.md
└── CMakeLists.txt

Build & Tech Stack

Languages

C++ (primary), C, Python (AerialVision), Lex/Yacc (parsers)

Build System

CMake + Make — supports both traditional Makefiles and modern CMake builds

Key Dependencies

CUDA Toolkit (4.2–11), Flex/Bison, Xlib (AerialVision), PTX grammar

Simulation Modes

Execution-driven (intercepts real CUDA calls) and Trace-driven (Accel-Sim SASS traces via NVBit)