cuda-sim — PTX Functional Simulation Engine

PTX Functional Simulation Engine

The cuda-sim module is the functional simulation core of GPGPU-Sim. It parses PTX/PTXPlus assembly into an internal IR, decodes 150+ instruction opcodes, and executes them per-thread inside warps and CTAs — modelling all CUDA memory spaces (global, shared, local, constant, texture). It provides the ground-truth architectural state that the timing model consults.

Files

24.5k

Lines of Code

15+

Classes

C++

Language

PTX ISA SIMT Execution PDOM Reconvergence Memory Spaces

What Is cuda-sim?

cuda-sim is the component that makes GPGPU-Sim actually run GPU programs. While the timing model worries about when things happen (pipeline stages, cache latencies), cuda-sim worries about what happens — computing the correct result for every thread, every instruction.

💡 Analogy — The Universal Translator

Think of cuda-sim as a universal translator for GPU programs. A CUDA kernel arrives as PTX text — a low-level assembly language. cuda-sim reads this foreign language, builds a phrase book (the IR), works out the grammar rules (CFG and post-dominators), and then faithfully interprets every sentence (instruction) for every speaker (thread) in the crowd (warp), making sure everyone arrives at the same meeting points (reconvergence) even when they take different conversational paths (divergence).

Architecture

flowchart LR
  subgraph Loading["PTX Loading"]
    L1["ptx_loader\n(ptx_loader.cc)"] --> L2["ptx_recognizer\n(ptx_parser.h)"]
  end
  subgraph IR["IR Construction"]
    L2 --> IR1["symbol_table"]
    L2 --> IR2["function_info"]
    L2 --> IR3["ptx_instruction"]
    L2 --> IR4["operand_info"]
  end
  subgraph Analysis["CFG & PDOM"]
    IR2 --> A1["basic_block_t\n(create CFG)"]
    A1 --> A2["Dominator\nAnalysis"]
    A2 --> A3["Post-Dominator\n& IPDOM"]
    A3 --> A4["pre_decode"]
  end
  subgraph Exec["Warp Execution"]
    A4 --> E1["functionalCoreSim\n(CTA loop)"]
    E1 --> E2["ptx_thread_info\n(per-lane exec)"]
    E2 --> E3["opcodes.def\nDispatch Switch"]
    E3 --> E4["instructions.cc\n(150+ impls)"]
  end
  subgraph Mem["Memory Model"]
    E4 --> M1["memory_space_impl\n(sparse pages)"]
    M1 --> M2["Global / Shared /\nLocal / Const / Tex"]
  end

Data Flow

sequenceDiagram
  participant Host as Host Runtime
  participant Loader as ptx_loader
  participant Parser as ptx_recognizer
  participant IR as function_info
  participant CFG as CFG Builder
  participant PDOM as PDOM Analyzer
  participant Core as functionalCoreSim
  participant Thread as ptx_thread_info
  participant Mem as memory_space_impl

  Host->>Loader: PTX string from binary
  Loader->>Parser: init_parser + ptx_parse()
  Parser->>IR: Build symbol_table, function_info, ptx_instruction
  IR->>CFG: create_basic_blocks + connect
  CFG->>PDOM: find_postdominators + find_ipostdominators
  PDOM->>IR: Annotate reconvergence PCs
  IR->>IR: ptx_assemble (assign PCs, resolve labels)
  IR->>IR: pre_decode (classify opcodes, set latency)
  Host->>Core: Launch kernel (CTA assignment)
  Core->>Core: initializeCTA (create threads, assign tid)
  loop Warp round-robin
    Core->>Thread: fetch instruction at PC
    Thread->>Thread: Evaluate predicate
    Thread->>Thread: Dispatch via opcodes.def switch
    Thread->>Mem: Read/write memory spaces
    Thread->>Thread: Update PC, collect stats
    Thread->>Core: updateSIMTStack (divergence)
  end
  Core->>Host: CTA complete

🚶 Step-by-Step Narrated Walkthrough

Follow a single add.f32 %f3, %f1, %f2 PTX instruction from binary loading through execution:

Binary Load — The host runtime calls gpgpu_ptx_sim_load_ptx_from_string() in ptx_loader.cc, passing the raw PTX text extracted from the CUDA fatbinary.
Lexer Init — init_parser() creates a Flex scanner via ptx_lex_init() and feeds the PTX string with ptx__scan_string().
Parsing — ptx_parse(scanner) invokes the Bison grammar. When it hits add.f32 %f3, %f1, %f2, the ptx_recognizer calls builder methods to construct an operand_info for each register and a ptx_instruction node with opcode ADD_OP, type .f32.
Symbol Resolution — %f1, %f2, %f3 are looked up (or created) in the hierarchical symbol_table. Each symbol records its type, register number, and enclosing scope.
Assembly — function_info::ptx_assemble() assigns a globally-unique PC to our instruction, populates the m_instr_mem[] array, and registers the instruction in g_pc_to_insn[].
CFG Construction — create_basic_blocks() groups sequential instructions into basic_block_t nodes; connect_basic_blocks() links them by successor/predecessor edges.
PDOM Analysis — find_postdominators() runs Muchnick’s iterative fixed-point algorithm over the CFG. find_ipostdominators() extracts the immediate post-dominator for each block — this becomes the reconvergence PC for branches.
Pre-decode — pre_decode() classifies ADD_OP as an integer/FP ALU op (classification 1), looks up its latency from -ptx_opcode_latency_fp (default: 1 cycle), and caches operand metadata.
Fetch — At runtime, functionalCoreSim::execute() iterates warps round-robin. For our warp, m_func_info->get_instruction(pc) returns the cached ptx_instruction* from m_instr_mem[pc - start].
Predicate Check — ptx_thread_info::ptx_exec_inst() first checks if the instruction has a predicate guard (@p). Our add is unconditional, so skip = false.
Dispatch — The macro-generated switch(inst_opcode) hits case ADD_OP:, which expands from opcodes.def to call add_impl(pI, this) in instructions.cc.
Operand Read — add_impl calls get_operand_value() for %f1 and %f2, reading their float values from the thread’s register file (m_regs.back()[reg_num]).
Compute & Write-back — The result f1 + f2 is stored via set_operand_value() into %f3. The PC advances by pI->inst_size(). Per-line stats are updated via ptx_stats.

Key Components

#	Class / Struct	File	Role
1	`cuda_sim`	cuda-sim.h	Top-level simulation context; owns config knobs, global lookup maps, kernel launch entry points
2	`functionalCoreSim`	cuda-sim.h	Executes one CTA functionally: creates warps, runs warp-round-robin loop, handles barriers
3	`ptx_thread_info`	ptx_sim.h	Per-thread execution state: register file, PC, call stack, memory accessors, instruction execution
4	`ptx_cta_info`	ptx_sim.h	Per-CTA bookkeeping: thread membership, barrier counter, exit tracking
5	`ptx_warp_info`	ptx_sim.h	Per-warp done-thread counter
6	`function_info`	ptx_ir.h	One PTX function/kernel: instruction memory, basic blocks, PDOM tree, parameters
7	`ptx_instruction`	ptx_ir.h	Single PTX instruction (extends `warp_inst_t`): opcode, operands, type, modifiers, pre-decode cache
8	`operand_info`	ptx_ir.h	Operand descriptor: register, immediate, memory, vector, builtin, with address-space annotation
9	`symbol` / `symbol_table`	ptx_ir.h	PTX symbol and hierarchical scope table; manages registers, variables, labels, type info
10	`basic_block_t`	ptx_ir.h	CFG node: predecessor, successor, dominator, and post-dominator sets
11	`memory_space`	memory.h	Abstract interface for sparse memory read/write with watchpoint support
12	`memory_space_impl<BSIZE>`	memory.h	Templated sparse-page memory model (block sizes 32 B – 16 KB)
13	`mem_storage<BSIZE>`	memory.h	Single memory page backed by `calloc`’d buffer
14	`ptx_recognizer`	ptx_parser.h	Flex/Bison parser state: scanner handle, directive and instruction builder methods
15	`ptxinfo_data`	ptx_loader.h	PTX loading pipeline: string → parse → assemble; ptxas integration
16	`ptx_stats`	ptx-stats.h	Per-source-line profiling: exec count, latency, DRAM traffic, bank conflicts, warp divergence
17	`ptx_reg_t`	ptx_sim.h	Union type for all PTX register widths: s8–s64, u8–u64, f16/f32/f64, pred, u128

Deep-Dive Topics

⚙ PDOM Reconvergence

How diverged warp threads rejoin at the immediate post-dominator.

Muchnick fixed-point algorithm
SIMT stack push/pop at branches
Reconvergence PC derivation

⚡ PTX Instruction Dispatch

Macro-driven switch over 150+ opcodes via opcodes.def.

OP_DEF / OP_W_DEF macros
Triple-include pattern (enum, switch, warp-sync)
Pipeline classification (1–11)

📋 Memory Space Model

Sparse paged memory for global, shared, local, constant, and texture spaces.

Hash-map page table (tr1_hash_map)
Block sizes: 32 B – 16 KB per space
Watchpoint-triggered notifications

🔄 Warp Execution Loop

Round-robin warp scheduling inside functionalCoreSim.

Per-lane ptx_exec_inst()
Barrier synchronisation (BAR_OP)
Thread lifecycle: NEW → RUNNING → DONE

📄 PTX Parsing Pipeline

From raw PTX text to assembled, pre-decoded instruction memory.

Flex/Bison scanner & grammar
Symbol table construction
Label resolution & PC assignment

Directory Map

Source tree for cuda-sim

src/cuda-sim/
├── cuda-sim.h / .cc            ← Top-level sim context, CTA execution loop, instruction dispatch
├── ptx_sim.h / .cc             ← ptx_thread_info, ptx_cta_info, ptx_reg_t, call stack
├── ptx_ir.h / .cc              ← function_info, ptx_instruction, operand_info, symbol_table, CFG, PDOM
├── ptx_loader.h / .cc          ← PTX string loading, Flex/Bison parser invocation, ptxas integration
├── ptx_parser.h                ← ptx_recognizer class, builder methods for directives & instructions
├── instructions.cc             ← 150+ opcode implementation functions (add_impl, mul_impl, ld_impl …)
├── memory.h / .cc              ← memory_space abstract interface, memory_space_impl<BSIZE>, mem_storage
├── opcodes.h / opcodes.def     ← Opcode enum generation & macro-driven dispatch table
├── ptx-stats.h / .cc           ← Per-source-line profiling (exec count, latency, bank conflicts)
├── ptx.l / ptx.y               ← Flex lexer & Bison grammar for PTX assembly language
├── cuda_device_runtime.h / .cc ← CUDA Dynamic Parallelism: device-side kernel launches
├── ptxinfo.l / ptxinfo.y       ← Parser for ptxas info output (register usage, spills)
├── decuda_pred_table/          ← Predicate truth tables for condition code evaluation
└── README                      ← Module notes

Configuration Knobs

#	Parameter	Default	Description
1	`-ptx_opcode_latency_int`	`1,1,19,25,145,32`	Integer ALU latencies: ADD, MAX, MUL, MAD, DIV, SHFL
2	`-ptx_opcode_latency_fp`	`1,1,1,1,30`	FP32 latencies: ADD, MAX, MUL, MAD, DIV
3	`-ptx_opcode_latency_dp`	`8,8,8,8,335`	FP64 latencies: ADD, MAX, MUL, MAD, DIV
4	`-ptx_opcode_latency_sfu`	`8`	SFU instruction latency (sin, cos, rcp, …)
5	`-ptx_opcode_latency_tensor`	`64`	Tensor-core (WMMA/MMA) instruction latency
6	`-ptx_opcode_initiation_int`	`1,1,4,4,32,4`	Integer initiation intervals
7	`-ptx_opcode_initiation_fp`	`1,1,1,1,5`	FP32 initiation intervals
8	`-ptx_opcode_initiation_dp`	`8,8,8,8,130`	FP64 initiation intervals
9	`-ptx_opcode_initiation_sfu`	`8`	SFU initiation interval
10	`-ptx_opcode_initiation_tensor`	`64`	Tensor initiation interval
11	`-cdp_latency`	`7200,8000,100,12000,1600`	CUDA Dynamic Parallelism API latencies
12	`-save_embedded_ptx`	`0`	Save PTX files embedded in binary as `<n>.ptx`
13	`-enable_ptx_file_line_stats`	`1`	Enable per-PTX-source-line profiling
14	`-ptx_line_stats_filename`	`gpgpu_inst_stats.txt`	Output file for source-line statistics

Environment Variables

Variable	Effect
`PTX_SIM_MODE_FUNC`	`1` = functional-only simulation, `0` = detailed performance sim
`PTX_SIM_DEBUG`	Set debug verbosity level (integer)
`PTX_SIM_DEBUG_THREAD_UID`	Restrict debug trace to one thread UID
`PTX_SIM_USE_PTX_FILE`	Override embedded PTX with external `.ptx` file
`CUDA_LAUNCH_BLOCKING`	Force synchronous kernel launches

Code Snippets

Macro-Generated Instruction Dispatch

// cuda-sim.cc — The heart of instruction execution
switch (inst_opcode) {
#define OP_DEF(OP, FUNC, STR, DST, CLASSIFICATION) \
  case OP:                                         \
    FUNC(pI, this);                                \
    op_classification = CLASSIFICATION;            \
    break;
#define OP_W_DEF(OP, FUNC, STR, DST, CLASSIFICATION) \
  case OP:                                           \
    FUNC(pI, get_core(), inst);                      \
    op_classification = CLASSIFICATION;              \
    break;
#include "opcodes.def"
#undef OP_DEF
#undef OP_W_DEF
  default:
    printf("Execution error: Invalid opcode (0x%x)\n", pI->get_opcode());
    break;
}

Per-Thread Instruction Execution Entry

// cuda-sim.cc — Entry point for executing one instruction on one thread
void ptx_thread_info::ptx_exec_inst(warp_inst_t &inst, unsigned lane_id) {
  addr_t pc = next_instr();
  assert(pc == inst.pc);  // timing ↔ functional sync check
  const ptx_instruction *pI = m_func_info->get_instruction(pc);
  set_npc(pc + pI->inst_size());

  bool skip = false;
  if (pI->has_pred()) {
    const operand_info &pred = pI->get_pred();
    ptx_reg_t val = get_operand_value(pred, pred, PRED_TYPE, this, 0);
    skip = (val.pred & 0x0001) ^ pI->get_pred_neg();
  }
  // ... dispatch switch (see above) ...
}

Sparse Paged Memory Model

// memory.h — Templated sparse memory with lazy page allocation
template <unsigned BSIZE>
class memory_space_impl : public memory_space {
  typedef mem_map<mem_addr_t, mem_storage<BSIZE>> map_t;
  map_t m_data;  // sparse page table (hash-map)

  void read(mem_addr_t addr, size_t length, void *data) const;
  void write(mem_addr_t addr, size_t length, const void *data,
             ptx_thread_info *thd, const ptx_instruction *pI);
};
// Instantiated for block sizes: 32, 64, 8192, 16384 bytes