AccelWattch — GPU Power Modeling Framework

GPU Power Modeling Framework

AccelWattch is the architectural power modeling component of GPGPU-Sim. Built on McPAT and CACTI, it estimates dynamic power, leakage power, and area for every major on-chip structure — cores, caches, register files, execution units, NoC, memory controllers, and I/O — by combining activity counters from cycle-level simulation with analytical circuit models. Validated against Volta-class GPUs.

Files

49.4k

Lines of Code

McPAT+CACTI

Foundation

Volta

Validated

Analytical Power Model CACTI Cache Modeling Dynamic+Leakage XML Configuration

💡 What is AccelWattch?

Analogy: Think of AccelWattch as an electricity bill estimator for a GPU factory. Imagine a massive factory (GPU chip) with assembly lines (SMs), warehouses (caches), conveyor belts (NoC), and loading docks (memory controllers). Each area has its own power meter. AccelWattch reads every meter every sampling interval, looks up the per-unit energy cost in a catalog (CACTI/McPAT technology tables), multiplies by how many operations each area performed, and produces a detailed bill — broken down by department — showing exactly where the watts are going.

It supports three modes: pure simulation (all counters from GPGPU-Sim), pure hardware (all from real GPU), and hybrid (selectively mixing real and simulated counters per component for validation).

Architecture

flowchart LR
  subgraph Input["Configuration Input"]
    XML["XML Config File"]
    PERF["Performance Counters\n(from GPGPU-Sim)"]
  end
  subgraph Parser["Parsing Layer"]
    XMLP["xmlParser\n(DOM tree)"]
    PXML["ParseXML\n(root_system struct)"]
  end
  subgraph Processor["Processor Hierarchy"]
    PROC["Processor"]
    CORE["Core\n(SM Model)"]
    SC["SharedCache\n(L2/L3)"]
    NOC["NoC\n(Interconnect)"]
    MC["MemoryController\n(DRAM)"]
    IO["I/O Controllers\n(NIC/PCIe/Flash)"]
  end
  subgraph Engine["CACTI Engine"]
    ARR["ArrayST\n(Array Solver)"]
    WIRE["Wire Model"]
    TECH["Technology DB\n(22-180nm ITRS)"]
  end
  subgraph Output["Power Output"]
    PWR["Per-Component\nPower Report"]
    TRACE["Power Trace\n(.gz files)"]
  end
  XML --> XMLP --> PXML --> PROC
  PERF --> PROC
  PROC --> CORE & SC & NOC & MC & IO
  CORE & SC --> ARR
  ARR --> WIRE --> TECH
  CORE & SC & NOC & MC & IO --> PWR --> TRACE

Data Flow

sequenceDiagram
  participant SIM as GPGPU-Sim
  participant PI as power_interface
  participant W as gpgpu_sim_wrapper
  participant P as Processor
  participant C as Core / Cache / MC
  participant CACTI as CACTI Engine
  participant OUT as Trace Files

  SIM->>PI: Every gpu_stat_sample_freq cycles
  PI->>PI: Collect perf counters (SMs, L2, DRAM, NoC)
  PI->>W: set_inst_power(), set_l1cache_power(), ...
  PI->>W: compute()
  W->>P: Processor::compute()
  P->>C: computeEnergy(is_tdp=false)
  C->>CACTI: accesses × energy_per_access
  CACTI-->>C: rt_power (dynamic + leakage)
  C-->>P: component power
  P-->>W: aggregated power
  W->>W: update_components_power()
  W->>W: power_metrics_calculations()
  W->>OUT: dump() → power_trace.gz

🚶 Step-by-Step Narrated Walkthrough

Follow a single power estimation cycle as it flows through the entire AccelWattch pipeline:

Sampling Trigger — The GPU simulator reaches a gpu_stat_sample_freq boundary (e.g., every 10,000 cycles). power_interface.cc::mcpat_cycle() fires.
Counter Collection — The bridge layer iterates over every shader_core and memory_partition_unit, harvesting counters: total instructions, cache hits/misses, register reads/writes, ALU/FPU/SFU accesses, DRAM reads/writes, and NoC flits.
Counter Injection — Each counter set is injected into gpgpu_sim_wrapper via 20+ typed set_*_power() methods (e.g., set_inst_power(), set_regfile_power(), set_exec_unit_power()). In hybrid mode, hardware counters may replace some simulated values.
XML Stats Update — The wrapper writes injected counters into the ParseXML struct tree (root_system → system_core[i] → total_instructions, etc.), overwriting the TDP defaults with actual runtime values.
Processor::compute() — The top-level Processor iterates over its component vectors: cores[], l2array[], l3array[], nocs[], mc, and I/O controllers, calling computeEnergy(false) on each.
Core::computeEnergy() — Each Core (SM model) updates sub-unit stats and delegates: InstFetchU calculates icache + BTB power, EXECU calculates register file + ALU/FPU/MUL + bypass power, LoadStoreU calculates dcache/shared-memory power. Each sub-unit calls ArrayST::computeEnergy() which multiplies access count by CACTI’s pre-computed energy-per-access.
CACTI Energy Lookup — For each array structure (icache, dcache, register file, TLB, BTB), the pre-solved uca_org_t provides energy per read/write/search operation. Runtime power = (accesses / total_cycles) × energy_per_access × clock_rate. Leakage is added from I_off × V_dd.
SharedCache & MC — L2 cache arrays similarly compute energy. The MemoryController computes power for the front-end command queue, back-end protocol engine, PHY transceiver, and DRAM die refresh.
NoC Power — The network-on-chip multiplies router/link energy by flit count, modulated by a traffic pattern factor (default 0.6).
Power Aggregation — update_components_power() extracts 33 named component powers (IBP, ICP, DCP, TCP, RFP, FPUP, L2CP, MCP, NOCP, DRAMP, PIPEP, etc.) into the sample_cmp_pwr[] vector.
Metrics Calculation — power_metrics_calculations() updates running average/max/min for each component and total GPU power across the kernel.
Trace Output — dump() writes the current sample to gzip-compressed trace files (power_trace.gz, metric_trace.gz). Steady-state detection checks if power variance has converged below steady_power_deviation.

Key Components

#	Component	File(s)	Purpose
1	`gpgpu_sim_wrapper`	gpgpu_sim_wrapper.h/.cc	Integration shim — bridges GPGPU-Sim perf counters to McPAT. Manages XML loading, counter injection, power computation, and trace I/O.
2	`Processor`	processor.h/.cc	Top-level hierarchy — owns all Cores, SharedCaches, NoCs, MemoryControllers, and I/O controllers. Runs TDP and runtime `computeEnergy()`.
3	`Core`	core.h/.cc	GPU Streaming Multiprocessor model (~290 KB). Contains InstFetchU, LoadStoreU, MemManU, EXECU, RENAMINGU, Pipeline, UndiffCore.
4	`ParseXML`	XML_Parse.h/.cc	XML configuration parser — reads config into nested `root_system` struct tree. Defines 44 performance counter enums.
5	`xmlParser`	xmlParser.h/.cc	Lightweight XML DOM — provides `XMLNode` tree with parsing, navigation, modification. Supports UTF-8/ShiftJIS/GB2312.
6	`SharedCache`	sharedcache.h/.cc	L2/L3 cache power model with unified arrays plus miss/fill/prefetch/WB buffers. Supports directory types (ST/SBT/DC).
7	`MemoryController`	memoryctrl.h/.cc	DRAM controller hierarchy: MCFrontEnd (queues), MCBackend (protocol), MCPHY (SerDes), DRAM (refresh). DDR2/3/GDDR3/5.
8	`NoC`	noc.h/.cc	Network-on-Chip — router-based or bus-based interconnect with VC buffers, crossbar, arbiter. Traffic-pattern modulated.
9	`interconnect`	interconnect.h/.cc	Wire/bus modeling — wraps CACTI Wire class. Supports global/semi-global/local/low-swing wires with auto-pipelining.
10	`ArrayST`	array.h/.cc	SRAM/CAM array wrapper over CACTI `uca_org_t`. `optimize_array()` iteratively relaxes timing for minimum-energy solution.
11	`InstFetchU`	core.h	Instruction fetch unit: icache, MSHR, instruction fill buffer, prefetch buffer, BranchPredictor, BTB, inst_decoder ×3.
12	`EXECU`	core.h	Execution unit: register file (IRF/FRF/OPC), scheduler (instruction window), FPU/ALU/MUL functional units, bypass interconnects.
13	`LoadStoreU`	core.h	Load-store unit: dcache, ccache (constant cache), tcache (texture cache), shared memory, LSQ, internal crossbar & NoC.
14	`FunctionalUnit`	logic.h/.cc	ALU/FPU/MUL power from CACTI area-based estimation scaled by operation type and duty cycle.
15	`Pipeline`	logic.h/.cc	Pipeline register power. `compute_stage_vector()` builds stage count; DFF-based register energy per stage.
16	`basic_components`	basic_components.h/.cc	Enums, parameter structs (`CoreDynParam`, `CacheDynParam`, `MCParam`, `NoCParam`), and utility functions.
17	`I/O Controllers`	iocontrollers.h/.cc	NIUController (Ethernet), PCIeController (multi-lane PHY), FlashController (NAND/SSD). Empirical area/power models.

Sub-Modules & Logical Groupings

cacti/

Cache Access and Cycle Time model (HP Labs). Analytical SRAM/DRAM/CAM array solver providing area, timing, and power.

Exhaustive partition-space search (Ndwl × Ndbl × Nspd)
22–180 nm ITRS technology database
Wire delay with repeater optimization
H-tree routing & NUCA mesh support

Core Components

GPU Streaming Multiprocessor power model — the largest component. Processor hierarchy and SM sub-units.

processor.h/.cc — top-level hierarchy
core.h/.cc — SM model (~290 KB)
logic.h/.cc — FU, pipeline, decoders
array.h/.cc — ArrayST CACTI bridge

Memory Controllers

DRAM controller hierarchy and shared cache models for L2/L3 with directory support.

memoryctrl.h/.cc — MC front/backend/PHY/DRAM
sharedcache.h/.cc — L2/L3 unified arrays
DDR2/3, GDDR3/5 protocol support

Interconnect / NoC

On-chip network and wire/bus modeling with process-scaled delay and power.

noc.h/.cc — router or bus-based NoC
interconnect.h/.cc — CACTI Wire wrapper
VC buffers, crossbar, arbiter

I/O Controllers

Off-chip interface power models with empirical area/power scaling by process node.

iocontrollers.h/.cc — NIC, PCIe, Flash
NIUController: Ethernet MAC+PCS+SerDes
PCIeController: multi-lane PHY

XML Parsing

Configuration loading pipeline: low-level DOM parsing and high-level struct population.

xmlParser.h/.cc — lightweight XML DOM
XML_Parse.h/.cc — config → root_system
44 performance counter enums

Directory Map

Source tree for AccelWattch

src/accelwattch/
├── cacti/                         ← CACTI cache/memory analytical model (HP Labs)
│   ├── cacti_interface.h/.cc      ← Public API: InputParameter, uca_org_t, powerDef
│   ├── Ucache.h/.cc               ← Partition-space solver (exhaustive search)
│   ├── bank.h/.cc                 ← Bank-level aggregation of Mat objects
│   ├── mat.h/.cc                  ← Subarray timing model (~82 KB)
│   ├── subarray.h/.cc             ← Cell array physical dimensions
│   ├── parameter.h/.cc            ← DynamicParameter + InputParameter validation
│   ├── wire.h/.cc                 ← Wire delay/power with repeater insertion
│   ├── area.h/.cc                 ← Area accounting (height × width)
│   ├── decoder.h/.cc              ← Row/column decoders (logical effort)
│   ├── htree2.h/.cc               ← H-tree signal distribution
│   ├── uca.h/.cc                  ← Unified Cache Architecture combiner
│   ├── nuca.h/.cc                 ← Non-Uniform Cache Architecture (mesh)
│   ├── router.h/.cc               ← 2D mesh router model
│   ├── crossbar.h/.cc             ← N×M tri-state buffer switch
│   ├── arbiter.h/.cc              ← VC arbitration logic
│   ├── basic_circuit.h/.cc        ← Transistor-level primitives
│   ├── component.h/.cc            ← Base Component class
│   ├── const.h                    ← Array size limits, enums
│   ├── technology.cc              ← ~130 KB ITRS device/wire parameters
│   └── io.h/.cc                   ← Cache config parsing and result output
├── gpgpu_sim_wrapper.h/.cc        ← Integration shim (GPGPU-Sim ↔ McPAT)
├── processor.h/.cc                ← Top-level component hierarchy
├── core.h/.cc                     ← GPU SM model (largest file, ~290 KB)
├── XML_Parse.h/.cc                ← XML config → root_system struct
├── xmlParser.h/.cc                ← Lightweight XML DOM parser
├── sharedcache.h/.cc              ← L2/L3 cache power model
├── memoryctrl.h/.cc               ← DRAM controller hierarchy
├── noc.h/.cc                      ← Network-on-Chip model
├── interconnect.h/.cc             ← Wire/bus modeling (CACTI Wire)
├── logic.h/.cc                    ← FunctionalUnit, Pipeline, decoders
├── basic_components.h/.cc         ← Parameter structs and enums
├── array.h/.cc                    ← ArrayST CACTI bridge + optimize_array()
├── iocontrollers.h/.cc            ← NIC / PCIe / Flash controllers
├── arch_const.h                   ← Default architecture constants (90nm)
├── globalvar.h                    ← Global opt_for_clk flag
├── technology_xeon_core.cc        ← Technology init for 22–90nm nodes
└── main.cc                        ← Standalone McPAT entry point

Configuration Knobs

Parameter	Source	Example	Description
`number_of_cores`	XML → system	30	Streaming Multiprocessor count
`number_of_L2s`	XML → system	4	L2 cache partition count
`core_tech_node`	XML → system	45	Technology node in nanometers
`target_core_clockrate`	XML → system	1300	Target frequency (MHz)
`temperature`	XML → system	380	Junction temperature (Kelvin)
`device_type`	XML → system	0	0=High Performance, 1=LSTP, 2=LOP
`machine_type`	XML → core	1	0=Out-of-Order, 1=In-order (GPU)
`pipeline_depth`	XML → core	6,6	Integer and FP pipeline stages
`icache_config`	XML → core.icache	16384,32,4,1,1,3	Size, block, assoc, banks, throughput, latency
`L2_config`	XML → L2	786432,64,16,1,1,23	L2 cache geometry (same format)
`power_simulation_enabled`	gpgpusim.config	1	Enable/disable AccelWattch power model
`power_simulation_mode`	gpgpusim.config	0	0=sim only, 1=HW only, 2=hybrid
`gpu_stat_sample_freq`	gpgpusim.config	10000	Power sampling interval (cycles)
`steady_power_deviation`	gpgpusim.config	0.01	Convergence threshold for steady-state detection
`accelwattch_hybrid_configuration[]`	gpgpusim.config	30+ flags	Per-component HW/sim selection for hybrid mode

Code Snippets

Integration API — gpgpu_sim_wrapper

// gpgpu_sim_wrapper.h (simplified)
class gpgpu_sim_wrapper {
public:
  gpgpu_sim_wrapper(bool power_simulation_enabled, char* xmlfile,
                    int power_simulation_mode, bool dvfs_enabled);

  // Performance counter injection (called each sampling interval)
  void set_inst_power(bool clk_gated_lanes, double tot_cycles,
                      double busy_cycles, double tot_inst, ...);
  void set_regfile_power(double reads, double writes, double ops);
  void set_exec_unit_power(double fpu_acc, double ialu_acc, double sfu_acc);
  void set_mem_ctrl_power(double reads, double writes, double dram_precharge);

  // Trigger power calculation and output
  void compute();
  void update_components_power();
  void power_metrics_calculations();
  void dump();

private:
  Processor* proc;       // McPAT processor model
  ParseXML* p;           // Parsed XML configuration
};

Component Hierarchy — Processor

// processor.h (simplified)
class Processor : public Component {
public:
  vector<Core *> cores;
  vector<SharedCache *> l2array, l3array;
  vector<NoC *> nocs;
  MemoryController *mc;
  NIUController *niu;
  PCIeController *pcie;

  Processor(ParseXML *XML_interface);
  void compute();                  // Runtime power from activity counters
  void displayEnergy(uint32_t indent, int plevel, bool is_tdp);

  // Power coefficient extraction for linear model
  double get_const_dynamic_power();
  double get_coefficient_l2_read_hits();
  void coefficient_scale();
};

CACTI Array Bridge — ArrayST

// array.h (simplified) — iterative timing relaxation
class ArrayST : public Component {
public:
  InputParameter l_ip;
  uca_org_t local_result;    // CACTI solution (timing, area, power)

  ArrayST(const InputParameter* ip, string name,
          enum Device_ty dev, bool opt_local = true);

  // Relax timing constraints to find minimum-energy configuration
  virtual void optimize_array() {
    while ((throughput_overflow || latency_overflow)
           && cycle_time_dev > 10) {
      compute_base_power();     // Calls CACTI solver
      cycle_time_dev -= 10;     // Relax by 10%
      if (meets_timing) candidates.push_back(local_result);
    }
    local_result = *min_energy(candidates);
  }

  void leakage_feedback(double temperature);
};