InterSim2 — Interconnection Network Simulator

Cycle-Accurate On-Chip Network Simulation for GPGPU-Sim

InterSim2 is a cycle-accurate interconnection network (NoC) simulator integrated into GPGPU-Sim, based on Stanford’s BookSim 2.0. It models flit-level routing, virtual-channel flow control, switch allocation, and crossbar traversal across configurable topologies — connecting shader cores to memory partitions with faithful timing.

Sub-Modules

115

Files

26.6k

Lines of Code

Topologies

BookSim-based Virtual-Channel Router Credit Flow Control Cycle-Accurate

💡 Think of it like a Highway System

Imagine the GPU die as a city. Shader cores (SMs) are factories producing goods (memory requests), and memory partitions are warehouses. InterSim2 is the highway network connecting them. Each flit is a delivery truck, routers are highway interchanges, virtual channels are lane markings that prevent a stalled truck from blocking the entire road, and credits are “space available” signs that tell upstream interchanges when to send the next truck. The allocators are traffic lights deciding which truck goes through an intersection each cycle.

Architecture

flowchart LR
  subgraph GPGPU["GPGPU-Sim"]
    ICI["InterconnectInterface\n(Push / Pop / Advance)"]
  end

  subgraph TM["Traffic Management"]
    GTM["GPUTrafficManager"]
    TMP["TrafficManager\n(base)"]
    GTM --> TMP
  end

  subgraph Routers["routers/"]
    IQR["IQRouter\n(RC→VA→SA→ST)"]
    CR["ChaosRouter"]
    ER["EventRouter"]
  end

  subgraph Allocators["allocators/"]
    ISL["iSLIP"]
    WF["Wavefront"]
    SEP["Separable\n(In/Out First)"]
    PIM_A["PIM"]
  end

  subgraph Arbiters["arbiters/"]
    RR["RoundRobin"]
    MAT["Matrix"]
    PRI["Priority"]
    TRE["Tree"]
  end

  subgraph Networks["networks/"]
    MESH["KNCube\n(Mesh/Torus)"]
    FAT["FatTree"]
    CM["CMesh"]
    FF["FlatFly"]
    AN["AnyNet"]
  end

  subgraph Power["power/"]
    PM["Power_Module"]
    SM["SwitchMonitor"]
    BM["BufferMonitor"]
  end

  ICI -->|"_GeneratePacket()"| GTM
  GTM -->|"_Inject()"| Networks
  Networks -->|"ReadInputs()"| Routers
  Routers -->|"VA request"| Allocators
  Allocators -->|"Arbitrate()"| Arbiters
  Routers -->|"activity"| Power
  Routers -->|"WriteOutputs()"| Networks

Data Flow — Flit Lifecycle

sequenceDiagram
  participant GPU as GPGPU-Sim
Memory System
  participant ICI as InterconnectInterface
  participant GTM as GPUTrafficManager
  participant Net as Network Topology
  participant Rtr as IQRouter
  participant Alloc as Allocators
  participant Dest as Destination Router
  participant Ejc as Ejection Buffer

  GPU->>ICI: Push(deviceID, data, size)
  ICI->>GTM: _GeneratePacket() → create flits
  GTM->>Net: _Inject() → write to input channel
  Net->>Rtr: ReadInputs() → flit arrives
  Rtr->>Rtr: RC: Route Computation
  Rtr->>Alloc: VA: VC Allocation request
  Alloc-->>Rtr: VC granted
  Rtr->>Alloc: SA: Switch Allocation request
  Alloc-->>Rtr: Crossbar port granted
  Rtr->>Rtr: ST: Switch Traversal
  Rtr->>Net: WriteOutputs() → flit on wire
  Note over Rtr,Dest: Credit sent upstream
  Net->>Dest: Flit arrives at destination
  Dest->>GTM: _RetireFlit() → WriteOutBuffer()
  GTM->>Ejc: Transfer2BoundaryBuffer()
  GPU->>ICI: Pop(deviceID) → retrieve data

🚶 Step-by-Step: Tracing a Flit Through the Network

Follow a single memory read request flit from a shader core to a memory partition:

Push (Injection) — GPGPU-Sim calls InterconnectInterface::Push(sm_id, mem_id, mem_fetch*, size). The device IDs are mapped to internal network node IDs via _node_map, and subnet 0 is selected (shader→memory direction).
Packetization — GPUTrafficManager::_GeneratePacket() splits the request into n_flits = ceil(size / flit_size) flit objects. The first is marked head=true, the last tail=true. The original mem_fetch* is attached to the head flit’s data pointer.
Input Queue — Flits are pushed to _input_queue[subnet][node][class]. During the next _Step(), the _Inject() method moves them into _partial_packets and writes the head flit onto the network injection channel.
Channel Traversal (Source→First Router) — The flit enters a FlitChannel with configurable latency. After the channel delay, the flit is available at the first router’s input port via ReadInputs().
Route Computation (RC) — IQRouter::_RouteEvaluate() calls the topology’s routing function (e.g., XY for mesh) on the head flit. The result is an OutputSet listing valid output ports and VC ranges. The VC transitions from idle → routing.
VC Allocation (VA) — _VCAllocEvaluate() submits the head flit’s request to the VC allocator (e.g., iSLIP). The allocator matches input VCs to available output VCs. On success, VC state advances to vc_alloc → active.
Switch Allocation (SA) — _SWAllocEvaluate() requests a crossbar input→output connection. The switch allocator (e.g., iSLIP/Wavefront) resolves contention among all VCs competing for the same output. The router checks BufferState::IsFullFor(vc) to ensure downstream credits are available.
Switch Traversal (ST) — _SwitchEvaluate() moves the flit across the crossbar into the output buffer. For multi-flit packets, hold_switch_for_packet can keep the crossbar path reserved.
Credit Return — As the flit departs the input buffer, a Credit object is sent back upstream on the credit channel. The upstream router’s BufferState::ProcessCredit() increments its credit counter, allowing future injections.
Hop-by-Hop Repetition — Steps 4–9 repeat at each intermediate router along the route. For a 4×4 mesh with XY routing, a corner-to-corner packet traverses up to 6 hops.
Ejection — At the destination router, the flit is written to the ejection channel. TrafficManager::_RetireFlit() collects latency statistics and calls WriteOutBuffer().
Boundary Buffer Reassembly — Transfer2BoundaryBuffer() moves the flit into _boundary_buffer[subnet][node][vc]. The interface waits for all flits (head through tail) before marking the packet as complete.
Pop (Delivery) — GPGPU-Sim calls InterconnectInterface::Pop(mem_id). The interface round-robins across VCs, finds the reassembled packet, and returns the original void* (mem_fetch*) pointer to the memory partition.

Key Components

#	Class	File(s)	Role
1	`InterconnectInterface`	interconnect_interface.hpp/cpp	Primary entry point bridging GPGPU-Sim ↔ BookSim. `Push()` injects, `Pop()` retrieves, `Advance()` steps.
2	`GPUTrafficManager`	gputrafficmanager.hpp/cpp	GPU-specific traffic manager. Creates flits from `mem_fetch*`, manages per-subnet input queues.
3	`TrafficManager`	trafficmanager.hpp/cpp	Base orchestrator: injection, network stepping, flit retirement, statistics collection.
4	`Flit`	flit.hpp/cpp	Fundamental data unit. Types: READ_REQUEST, READ_REPLY, WRITE_REQUEST, WRITE_REPLY. Pool-allocated.
5	`Credit`	credit.hpp/cpp	Flow-control token carrying freed VC set. Pool-allocated.
6	`Router`	router.hpp	Abstract router base: `ReadInputs()`, `Evaluate()`, `WriteOutputs()`. Factory: `NewRouter()`.
7	`IQRouter`	iq_router.hpp/cpp	Primary input-queued VC router. Pipeline: RC → VA → SA → ST with speculative allocation.
8	`Network`	network.hpp	Abstract topology base: `_ComputeSize()`, `_BuildNet()`, `WriteFlit()`, `ReadFlit()`.
9	`Buffer`	buffer.hpp/cpp	Input buffer holding a vector of VC objects per router port.
10	`BufferState`	buffer_state.hpp/cpp	Downstream credit tracker with pluggable buffer sharing policies.
11	`VC`	vc.hpp/cpp	Virtual channel state machine: idle → routing → vc_alloc → active.
12	`Channel<T>`	channel.hpp	Template pipelined channel with configurable latency.
13	`FlitChannel`	flitchannel.hpp/cpp	Specialized channel for flits with per-class activity counters for power modeling.
14	`OutputSet`	outputset.hpp/cpp	Routing result: set of {output_port, vc_start, vc_end, priority} entries.
15	`Allocator`	allocator.hpp	Abstract allocator base with factory `NewAllocator()`. Dense and Sparse variants.
16	`Arbiter`	arbiter.hpp	Abstract arbiter base: `AddRequest()`, `Arbitrate()`. RoundRobin, Matrix, Priority, Tree.
17	`BookSimConfig`	booksim_config.hpp/cpp	Configuration store with all BookSim default parameters.
18	`IntersimConfig`	intersim_config.hpp/cpp	Extends BookSimConfig with GPU-specific knobs: `flit_size`, `perfect_icnt`, buffer sizes.
19	`Power_Module`	power/power_module.hpp	Analytical power model: wire, buffer, crossbar, and leakage power estimation.

Sub-Modules

allocators/

Switch & VC allocation algorithms that resolve contention when multiple VCs compete for crossbar ports.

iSLIP iterative round-robin matching
Wavefront diagonal-sweep matching
Separable (input-first / output-first)
PIM, LOA, MaxSize, SelAlloc

arbiters/

Low-level arbitration primitives that select one winner from competing requests on a single resource.

RoundRobin — rotating priority
Matrix — NxN priority with fair convergence
Priority — strict priority
Tree — hierarchical for scalability

routers/

Router microarchitectures implementing the pipeline that moves flits through the network.

IQRouter — primary VC router (RC→VA→SA→ST)
ChaosRouter — adaptive deflection routing
EventRouter — event-driven simulation
Speculative allocation & NOQ support

networks/

Network topology definitions — each computes router/channel counts and builds the interconnect graph.

KNCube (mesh & torus)
FatTree, CMesh, FlatFly
Butterfly, Dragonfly, AnyNet
10 topologies with per-topology routing

power/

Power and activity monitoring — tracks buffer/crossbar utilization for dynamic and leakage power estimation.

Power_Module — analytical power model
SwitchMonitor — crossbar activity factors
BufferMonitor — buffer R/W activity
Technology file parameterization

Directory Map

Source tree for InterSim2

src/intersim2/
├── allocators/                    ← Switch & VC allocation algorithms
│   ├── allocator.hpp/cpp          ← Base class + Dense/Sparse variants
│   ├── islip.hpp/cpp              ← iSLIP iterative round-robin
│   ├── wavefront.hpp/cpp          ← Wavefront diagonal sweep
│   ├── separable.hpp/cpp          ← Separable base
│   ├── separable_input_first.*    ← Input-first separable
│   ├── separable_output_first.*   ← Output-first separable
│   ├── pim.hpp/cpp                ← Parallel Iterative Matching
│   ├── loa.hpp/cpp                ← Lonely Output Allocator
│   ├── maxsize.hpp/cpp            ← Maximum-size matching
│   └── selalloc.hpp/cpp           ← Selection-based allocator
├── arbiters/                      ← Arbiter primitives
│   ├── arbiter.hpp/cpp            ← Base class
│   ├── roundrobin_arb.hpp/cpp     ← Round-robin arbiter
│   ├── matrix_arb.hpp/cpp         ← Matrix arbiter
│   ├── prio_arb.hpp/cpp           ← Strict-priority arbiter
│   └── tree_arb.hpp/cpp           ← Tree-structured arbiter
├── routers/                       ← Router microarchitectures
│   ├── router.hpp/cpp             ← Base class + factory
│   ├── iq_router.hpp/cpp          ← Input-Queued VC router (primary)
│   ├── chaos_router.hpp/cpp       ← Chaos/deflection router
│   └── event_router.hpp/cpp       ← Event-driven router
├── networks/                      ← Network topologies
│   ├── network.hpp/cpp            ← Base class
│   ├── kncube.hpp/cpp             ← Mesh & torus
│   ├── fattree.hpp/cpp            ← Fat-tree
│   ├── cmesh.hpp/cpp              ← Concentrated mesh
│   ├── flatfly_onchip.hpp/cpp     ← Flattened butterfly
│   ├── fly.hpp/cpp                ← Butterfly
│   ├── anynet.hpp/cpp             ← Arbitrary topology from file
│   ├── dragonfly.hpp/cpp          ← Dragonfly
│   ├── qtree.hpp/cpp              ← Quad-tree
│   └── tree4.hpp/cpp              ← 4-ary tree
├── power/                         ← Power modeling
│   ├── power_module.hpp/cpp       ← Top-level power model
│   ├── switch_monitor.hpp/cpp     ← Crossbar activity tracker
│   └── buffer_monitor.hpp/cpp     ← Buffer activity tracker
├── interconnect_interface.hpp/cpp ← GPGPU-Sim bridge (Push/Pop/Advance)
├── gputrafficmanager.hpp/cpp      ← GPU traffic manager
├── trafficmanager.hpp/cpp         ← Base traffic manager
├── flit.hpp/cpp                   ← Flit data structure
├── credit.hpp/cpp                 ← Credit flow-control token
├── buffer.hpp/cpp                 ← Input buffer (vector of VCs)
├── buffer_state.hpp/cpp           ← Downstream credit tracker
├── vc.hpp/cpp                     ← Virtual channel state machine
├── channel.hpp                    ← Template pipelined channel
├── flitchannel.hpp/cpp            ← Flit channel specialization
├── outputset.hpp/cpp              ← Routing result container
├── booksim_config.hpp/cpp         ← BookSim configuration
├── intersim_config.hpp/cpp        ← GPU-specific configuration
├── config_utils.hpp/cpp           ← Config file parser
├── module.hpp/cpp                 ← Base simulation module
├── timed_module.hpp               ← Cycle-based module interface
├── stats.hpp/cpp                  ← Statistics accumulator
├── traffic.hpp/cpp                ← Traffic pattern generators
├── injection.hpp/cpp              ← Injection process models
├── routefunc.hpp/cpp              ← Routing function registry
├── pipefifo.hpp                   ← Multi-lane pipeline FIFO
└── globals.hpp                    ← Global declarations

Configuration Knobs

Parameter	Default	Category	Description
`topology`	`"torus"`	Network	Network topology: torus, mesh, cmesh, fly, fattree, flatfly_onchip, anynet, dragonfly, qtree, tree4
`k`	8	Network	Network radix (nodes per dimension)
`n`	2	Network	Network dimension count
`network_count`	2	Network	Independent networks (typically 2: shader→mem, mem→shader)
`router`	`"iq"`	Router	Router type: iq (input-queued), chaos, event
`num_vcs`	16	Router	Virtual channels per physical port
`vc_buf_size`	8	Router	Buffer slots per virtual channel
`vc_allocator`	`"islip"`	Allocation	VC allocator: islip, separable_input_first, wavefront, pim, loa, maxsize
`sw_allocator`	`"islip"`	Allocation	Switch allocator algorithm (same options as vc_allocator)
`routing_function`	`"none"`	Routing	Routing algorithm (topology-dependent: dim_order, xy, romm, valiant, etc.)
`flit_size`	32	GPU	Flit size in bytes
`perfect_icnt`	0	GPU	Bypass NoC entirely (zero-latency ideal interconnect)
`input_buffer_size`	0 (=9)	GPU	Injection buffer capacity at the InterconnectInterface
`boundary_buffer_size`	16	GPU	Boundary buffer capacity for packet reassembly
`speculative`	0	Allocation	Enable speculative switch allocation (SA in parallel with VA)
`sim_power`	0	Power	Enable power estimation via activity monitoring

Code Highlights

Flit — The Fundamental Data Unit

class Flit {
public:
  enum FlitType { READ_REQUEST=0, READ_REPLY=1,
                  WRITE_REQUEST=2, WRITE_REPLY=3, ANY_TYPE=4 };
  FlitType type;
  int vc, cl;
  bool head, tail;
  int ctime, itime, atime;       // creation, injection, arrival times
  unsigned long long id, pid;     // flit ID, packet ID
  int src, dest, pri, hops;
  void* data;                     // payload pointer (e.g., mem_fetch*)
  OutputSet la_route_set;         // lookahead routing result

  static Flit * New();            // pool allocator
  void Free();
};

VC State Machine

class VC : public Module {
public:
  // idle → routing → vc_alloc → active → (tail departs) → idle
  enum eVCState { idle, routing, vc_alloc, active };

  void AddFlit(Flit *f);
  Flit *RemoveFlit();
  void SetState(eVCState s);
  void Route(tRoutingFunction rf, const Router* router,
             const Flit* f, int in_channel);
  void SetOutput(int port, int vc);
private:
  deque<Flit *> _buffer;
  eVCState _state;
  OutputSet *_route_set;
  int _out_port, _out_vc;
};

InterconnectInterface — GPGPU-Sim Bridge

class InterconnectInterface {
public:
  static InterconnectInterface* New(const char* config_file);
  void CreateInterconnect(unsigned n_shader, unsigned n_mem);
  void Push(unsigned input, unsigned output, void* data, unsigned size);
  void* Pop(unsigned output_deviceID);
  void Advance();       // steps the entire NoC one cycle
  bool Busy() const;    // any flits in-flight?
  bool HasBuffer(unsigned deviceID, unsigned size) const;
protected:
  vector<vector<vector<_BoundaryBufferItem>>> _boundary_buffer;
  vector<vector<vector<_EjectionBufferItem>>> _ejection_buffer;
  GPUTrafficManager* _traffic_manager;
  vector<Network *> _net;
};

How InterSim2 Connects to GPGPU-Sim Core

The sole interface is InterconnectInterface, accessed via the global g_icnt_interface. Shader cores call Push() to inject memory requests and Pop() at memory partitions to retrieve replies. The reverse path works identically on subnet 1.

sequenceDiagram
    participant SM as Shader Core (SM)
    participant ICI as InterconnectInterface
    participant NoC as InterSim2 Network
    participant MEM as Memory Partition

    SM->>ICI: Push(sm_id, mem_id, mem_fetch*, size)
    ICI->>NoC: Inject flits on subnet 0
    NoC-->>NoC: Route through mesh (multi-hop)
    NoC->>ICI: Eject at destination
    ICI->>MEM: Pop(mem_id) → mem_fetch*
    Note over MEM: Process memory request
    MEM->>ICI: Push(mem_id, sm_id, reply*, size)
    ICI->>NoC: Inject flits on subnet 1
    NoC-->>NoC: Route back through mesh
    NoC->>ICI: Eject at source
    ICI->>SM: Pop(sm_id) → reply*