mem — Memory System

The complete memory subsystem of gem5: ports, packets, caches, coherence protocols, DRAM controllers, and Memory Policies | gem5 Wiki" href="qos/index.html">QoS policies.

Sub-modules

750

Files

Classic + Ruby

Cache Hierarchy | gem5 Wiki" href="cache/index.html">Cache Models

C++

Language

12+

Replacement Policies

14+

Prefetchers

Table of Contents

Overview

Analogy

Think of the gem5 memory system as a postal service. A CPU generates a Packet (letter), which travels through Ports (mailboxes). The letter may be intercepted by a Cache (a local post office that keeps copies of popular items), or it may travel all the way to DRAM (the central warehouse). The Ruby system adds a full coherence protocol network — like a fleet of delivery trucks following traffic rules to prevent duplicate deliveries.

The src/mem/ directory contains all memory-related simulation models in gem5. It spans from the low-level Packet and Port abstractions through full cache hierarchies (Classic and Ruby), DRAM/NVM controllers, QoS policies, and memory probes.

Architecture Diagram

Click to expand

flowchart LR
    CPU["CPU Core"]
    ICache["L1 ICache"]
    DCache["L1 DCache"]
    L2["L2 Cache"]
    Bus["Coherent XBar"]
    MC["Memory Controller"]
    DRAM["DRAM / HBM"]
    Ruby["Ruby Network"]
    Dir["Directory Controller"]
    QoS["QoS Policy"]

    CPU -->|"RequestPort"| ICache
    CPU -->|"RequestPort"| DCache
    ICache -->|"ResponsePort"| L2
    DCache -->|"ResponsePort"| L2
    L2 -->|"RequestPort"| Bus
    Bus -->|"ResponsePort"| MC
    MC --> QoS
    QoS --> DRAM

    CPU -.->|"Alternative"| Ruby
    Ruby --> Dir
    Dir --> DRAM

Data Flow: Memory Request Lifecycle

Click to expand

sequenceDiagram
    participant CPU
    participant L1 as L1 DCache
    participant L2 as L2 Cache
    participant XBar as Coherent XBar
    participant MC as MemCtrl
    participant DRAM

    CPU->>L1: sendReq(ReadReq pkt)
    L1->>L1: Tag lookup
    alt Cache Hit
        L1-->>CPU: sendResp(ReadResp)
    else Cache Miss
        L1->>L2: sendReq(ReadReq)
        L2->>L2: Tag lookup
        alt L2 Hit
            L2-->>L1: sendResp(ReadResp)
            L1->>L1: Allocate block
            L1-->>CPU: sendResp(ReadResp)
        else L2 Miss
            L2->>XBar: sendReq(ReadReq)
            XBar->>MC: forward(ReadReq)
            MC->>DRAM: access()
            DRAM-->>MC: data
            MC-->>XBar: sendResp(ReadResp)
            XBar-->>L2: sendResp(ReadResp)
            L2->>L2: Allocate block
            L2-->>L1: sendResp(ReadResp)
            L1->>L1: Allocate block
            L1-->>CPU: sendResp(ReadResp)
        end
    end

Narrated Walkthrough: Cache Miss Request

Follow a ReadReq packet from the CPU through to DRAM and back:

CPU issues load — The CPU pipeline creates a Packet with command ReadReq and calls sendTimingReq() on its data port.

L1 DCache receives request — The CpuSidePort::recvTimingReq() handler in BaseCache begins processing.

Tag lookup — The cache checks its tag array. The CacheBlk is looked up via the indexing policy (e.g., set-associative).

Miss detected — No valid block found. The cache allocates an MSHR (Miss Status Holding Register) to track the outstanding request.

Forward to L2 — The L1 sends the request downstream via its MemSidePort using sendTimingReq().

L2 tag lookup — Same process repeats. If the L2 also misses, another MSHR is allocated.

Coherent XBar routing — The crossbar receives the request and routes it to the appropriate memory controller based on address ranges.

Memory controller queuing — The MemCtrl places the request in its read queue. QoS policies may reorder requests based on priority.

DRAM access — The controller issues the DRAM command sequence: activate row, read column, precharge. Timing is modeled cycle-accurately.

Response propagation — Data travels back: DRAM → MemCtrl → XBar → L2 → L1, each level filling its cache block.

MSHR deallocation — As each cache level receives the response, it deallocates its MSHR and unblocks any waiting requests for the same block.

CPU receives data — The L1 sends the ReadResp back to the CPU via sendTimingResp(), completing the memory access.

Key Components

Class / File	Header	Description
`Packet`	packet.hh	Core data transfer unit carrying command, address, data, and flags across the memory system.
`MemCmd`	packet.hh	Enumeration of all memory commands: ReadReq, WriteReq, WritebackDirty, Snoop*, etc.
`Request`	request.hh	Encapsulates the original CPU request with virtual/physical addresses and flags.
`RequestPort`	port.hh	Initiator-side port that sends requests and receives responses.
`ResponsePort`	port.hh	Target-side port that receives requests and sends responses.
`BaseCache`	cache/base.hh	Abstract base for all classic caches; manages tags, MSHRs, write buffers.
`Cache`	cache/cache.hh	Coherent cache supporting snooping, invalidation, and flexible topologies.
`NoncoherentCache`	cache/noncoherent_cache.hh	Simplified cache without coherence protocol — used for instruction caches or accelerators.
`MSHR`	cache/mshr.hh	Miss Status Holding Register — tracks pending cache misses and coalesces duplicate requests.
`AbstractMemory`	abstract_mem.hh	Base class for all memory backends (SimpleMemory, DRAM, NVM).
`MemCtrl`	mem_ctrl.hh	Main memory controller with read/write queuing, scheduling, and timing.
`DRAMInterface`	dram_interface.hh	Cycle-accurate DRAM timing model for DDR3/DDR4/DDR5/LPDDR/HBM.
`HBMCtrl`	hbm_ctrl.hh	Dual-channel HBM controller leveraging pseudo-channel architecture.
`CommMonitor`	comm_monitor.hh	Transparent probe inserted between ports to gather statistics.
`PacketQueue`	packet_queue.hh	Queue of packets with scheduling logic for deferred sends.

Sub-module Deep Dives

cache

Classic cache hierarchy: BaseCache, Cache, NoncoherentCache, MSHR, write buffers, replacement policies (LRU, BIP, BRRIP, SHiP, TreePLRU, …), prefetchers (Stride, BOP, AMPM, SPP, …), compressors, and tag organizations.

26 files · 4 sub-dirs (tags, replacement_policies, prefetch, compressors)

ruby

Protocol-driven coherence simulation using SLICC. Includes network models (Atomic and Timing Simple CPUs — gem5 Wiki" href="../cpu_doc/simple/index.html">simple, Garnet), sequencers, directory controllers, message buffers, GPU coalescer, and HTM support.

5 sub-dirs (system, network, structures, slicc_interface, protocol)

qos

Quality of Service policies for memory controllers: fixed-priority, proportional-fair scheduling, turnaround policies, and queue management for multi-tenant workloads.

21 files · FixedPrio + PF policies

probes

Memory probe infrastructure: MemTraceProbe for trace generation, StackDistProbe for reuse-distance analysis, MemFootprintProbe for working-set tracking.

4 headers · Plugs into any Port

protocol

Low-level protocol definitions: atomic.hh, functional.hh, timing.hh — the three transport interfaces that ports implement for different simulation modes.

3 headers

Controllers & Interfaces

DRAM controllers (MemCtrl, HBMCtrl, HeteroMemCtrl), external interfaces (DRAMSim2, DRAMSim3, DRAMSys), serial links, and the XBar system bridges.

15+ files · DDR3 / DDR4 / DDR5 / HBM / NVM

Directory Map

src/mem/ ├── packet.hh / packet.cc # Packet & MemCmd ├── port.hh / port.cc # RequestPort, ResponsePort ├── request.hh # Request class ├── abstract_mem.hh / .cc # AbstractMemory base ├── mem_ctrl.hh / .cc # Main memory controller ├── dram_interface.hh / .cc # DRAM timing model ├── hbm_ctrl.hh / .cc # HBM controller ├── hetero_mem_ctrl.hh / .cc # Heterogeneous memory ├── comm_monitor.hh / .cc # Communication monitor ├── packet_queue.hh / .cc # Packet queuing ├── cache/ # Classic cache hierarchy │ ├── base.hh, cache.hh # BaseCache, Cache │ ├── mshr.hh, write_queue.hh # Miss handling │ ├── replacement_policies/ # LRU, FIFO, BIP, BRRIP, … │ ├── prefetch/ # Stride, BOP, AMPM, SPP, … │ ├── tags/ # Set-assoc, FA-LRU, sector │ └── compressors/ # BDI, C-Pack, FPC, FPCD, … ├── ruby/ # Ruby coherence │ ├── system/ # Sequencer, RubySystem │ ├── network/ # Simple, Garnet networks │ ├── structures/ # CacheMemory, DirectoryMemory │ ├── slicc_interface/ # SLICC controllers │ └── protocol/ # Protocol definitions ├── qos/ # Quality of Service │ ├── mem_ctrl.hh # QoS MemCtrl │ ├── policy.hh # Policy base │ └── policy_fixed_prio.hh # Fixed-priority └── probes/ # Memory probes ├── mem_trace.hh # Trace probe └── stack_dist.hh # Stack distance

Configuration Reference

Parameter	System & Standard Library · gem5 Wiki" class="xref">SimObject	Description	Default
`size`	Cache	Total cache capacity	64kB (L1), 2MB (L2)
`assoc`	Cache	Set associativity	2 (L1), 8 (L2)
`tag_latency`	Cache	Tag array access cycles	2
`data_latency`	Cache	Data array access cycles	2
`response_latency`	Cache	Extra response delay	2
`mshrs`	Cache	Number of MSHRs	4
`write_buffers`	Cache	Write buffer entries	8
`replacement_policy`	Cache	Replacement policy object	LRURP
`prefetcher`	Cache	Prefetcher object	None
`mem_ranges`	AbstractMemory	Address ranges served	—
`device_size`	DRAMInterface	Per-device capacity	Varies by config
`tCK`	DRAMInterface	Clock period	Technology-dependent

Glossary

MSHR: Miss Status Holding Register — tracks outstanding cache misses and coalesces duplicate requests to the same block.
Snooping: A coherence mechanism where caches monitor (snoop) bus transactions to maintain data consistency.
SLICC: Specification Language for Implementing Cache Coherence — a domain-specific language used by Ruby to define protocols.
Garnet: A detailed on-chip network model within Ruby that models routers, links, and virtual channels.
XBar: Crossbar interconnect that routes packets between multiple ports based on address ranges.
QoS: Quality of Service — policies that prioritize or schedule memory requests based on requestor identity or priority.
Clusivity: Describes cache inclusion policy: inclusive (all L1 data in L2), exclusive (no duplication), or mostly-inclusive.