Request for Assistance Regarding gem5 HBMCtrl Issues

[LujhCoconut]

@mbabaie Thank you for your response to the issue I raised through email. Here is the specific and detailed information regarding the HBMCtrl issue.

This is a request for help on how to correctly configure and use HBMCtrl.
HBMCtrl is a subclass included in the latest version of gem5, located in the /src/mem directory and inheriting from MemCtrl.
HBMCtrl has two main attributes: one is dram, inherited from MemCtrl, and the other is dram_2. Refer to the specific description of HBMCtrl in the code files, where dram and dram_2 refer to the pseudo channels of HBM.

To correctly use HBMCtrl in SE Mode, I referred to the traditional DRAM (e.g. DDR4) memory configuration as MemCtrl.dram in SE Mode. Specifically, I referred to the MemConfig.py file in the /configs/common directory. In this configuration file, the most crucial code snippet is as following：

# ......
if opt_mem_type:
        intf = ObjectList.mem_list.get(opt_mem_type)
# ......
range_iter = 0
for r in system.mem_ranges:
        range_iter += 1
        for i in range(nbr_mem_ctrls):
            if opt_mem_type and (not opt_nvm_type or range_iter % 2 != 0):
                dram_intf = create_mem_intf(intf, r, i, intlv_bits, intlv_size, opt_xor_low_bit)
                if (issubclass(intf, m5.objects.DRAMInterface) and opt_mem_ranks):
                    dram_intf.ranks_per_channel = opt_mem_ranks
                if issubclass(intf, m5.objects.DRAMInterface):
                    dram_intf.enable_dram_powerdown = opt_dram_powerdown
                if opt_elastic_trace_en:
                    dram_intf.latency = "1ns"
                    print("For elastic trace, over-riding Simple Memory ""latency to 1ns."  )
                # Create the controller that will drive the interface
                mem_ctrl = dram_intf.controller()
                mem_ctrls.append(mem_ctrl)
# ......

The code above accepts the name of the DRAM passed through the command line, locates the corresponding memory medium in DRAMInterface.py based on this name, and uses the memory timing parameters and other information from the memory medium along with interleaving information to generate the appropriate interface, which is then assigned to MemCtrl.dram. After that, simply need to correctly connect MemCtrl to the membus to complete the configuration.
The above operation is not difficult to understand, but some issues seem to arise when configuring HBMCtrl, since HBMCtrl has two pseudo channels.

Here is a brief description of the configuration I aim to achieve.
The configuration I aim to achieve is actually quite simple: an 8GB HBM2 memory. The memory specifications are based on HBM_2000_4H_1x64. I need it to correctly interleave addresses to fully utilize the bandwidth of HBM2 during memory access.

The HBM_2000_4H_1x64 provided by DRAMInterface can be considered as a single pseudo channel, with a size of 256MB. Therefore, a single HBMCtrl represents one channel, which has a size of 512MB. Thus, when I attempt to configure an 8GB HBM2, I need 16 HBMCtrl instances.

Below is a detailed description of how I attempted to configure it
First, in SE mode, I provided an 8GB memory space as mem_range.

start = 0
end = 8*1024
mem_range = AddrRange(str(start) + 'MiB', str(end) + 'MiB')
system.mem_ranges.append(mem_range)

After some additional steps, I called HBMMemoryConfig.config_mem(options, system) to configure the memory. HBMMemoryConfig is a memory configuration file modified based on MemConfig.

HBMMemoryConfig.config_mem(options, system)

My first attempt is as follows:

    # .......
    nbr_hbm_ctrls = 16
    for r in system.mem_ranges:
        intlv_bits = int(math.log(nbr_hbm_ctrls, 2))
        if 2 ** intlv_bits != nbr_hbm_ctrls:
            fatal("Number of memory channels must be a power of 2")
        for i in range(nbr_hbm_ctrls):
            intf = ObjectList.mem_list.get(opt_hbm_type)
            hbm_intf = create_mem_intf(intf, r, i,
                                       intlv_bits, intlv_size, opt_xor_low_bit)
            hbm_ctrl = m5.objects.HBMCtrl()
            hbm_ctrl.dram = hbm_intf
            hbm_ctrl.dram_2 = hbm_intf
            mem_ctrls.append(hbm_ctrl)
      # .......

I must admit that my first attempt was quite foolish. I assigned the same interface to both HBMCtrl.dram and HBMCtrl.dram_2, which resulted in an error. The error message is as follows:

fatal: system.membus has two ports responding within range [0:0x200000000] ...............

After some consideration, I revised the configuration to enable interleaving across all pseudo channels as following:

# number of pseudo channels
    nbr_pch= 2 * nbr_hbm_ctrls
    for r in system.mem_ranges:
        intlv_bits = int(math.log(nbr_pch, 2))
        if 2 ** intlv_bits != nbr_pch:
            fatal("Number of memory channels must be a power of 2")
        j = 0
        # get the type of HBM
        intf = ObjectList.mem_list.get(opt_hbm_type)
        hbm_ctrl = m5.objects.HBMCtrl()
        # To enable interleaving across all pseudo channels, it is also necessary to correctly assign values to each channel.
        for i in range(nbr_pch):
            if i % 2 == 0:
                hbm_ctrl = m5.objects.HBMCtrl()
                intf_1 = create_mem_intf(intf, r, i,
                                intlv_bits, intlv_size, opt_xor_low_bit)
                j = i + 1
                intf_2 = create_mem_intf(intf, r, j,
                                intlv_bits, intlv_size, opt_xor_low_bit)
                hbm_ctrl.dram = intf_1
                hbm_ctrl.dram_2 = intf_2
                mem_ctrls.append(hbm_ctrl)

This configuration does not produce error messages at the start of running the SPEC 2017 benchmarks, so the issue with duplicate interfaces may have been resolved. However, during the continued execution of the SPEC 2017 benchmarks, it quickly results in a core dump. Specifically, the core dump occurs after the Memory Access phase of the SPEC 2017 benchmarks has completed all read and write operations at zero clocks.

For example, one of the core dump command line prompts provides the following error message:

src/mem/mem_ctrl.cc:633: panic: panic condition !mem_intr->getAddrRange().contains(pkt->getAddr()) occurred: Can't handle address range for packet ReadSharedReq [2a2080:2a20bf] IF

To better understand why the error message occurs, I added some DPRINTF statements for debugging in the relevant code sections.I used --debug-flag=MemCtrl to capture the debugging information.

The added debugging information is as follows:

void
MemCtrl::accessAndRespond(PacketPtr pkt, Tick static_latency,
                                                MemInterface* mem_intr)
{
    DPRINTF(MemCtrl, "Responding to Address %#x.. \n", pkt->getAddr());

    bool needsResponse = pkt->needsResponse();
    // do the actual memory access which also turns the packet into a
    // response
    DPRINTF(MemCtrl," getAddrRange ==== %llu - %llu\n ",mem_intr->getAddrRange().start(),mem_intr->getAddrRange().end());
    DPRINTF(MemCtrl," pkt_addr ==== %llu \n ",pkt->getAddr());
    DPRINTF(MemCtrl," panic_state ==== %d \n ",mem_intr->getAddrRange().contains(pkt->getAddr()));
    panic_if(!mem_intr->getAddrRange().contains(pkt->getAddr()),
             "Can't handle address range for packet %s\n", pkt->print());
    mem_intr->access(pkt);
// ......
}

The corresponding debugging information output is as follows:

  49000: system.mem_ctrls00: Responding to Address 0x2a2080.. 
  49000: system.mem_ctrls00:  getAddrRange ==== 0 - 8589934592
  49000: system.mem_ctrls00:  pkt_addr ==== 2760832 
  49000: system.mem_ctrls00:  panic_state ==== 0 

According to the code snippet above, the address of the packet, pkt_addr, is 2,760,832 in decimal, while the memory interface address range is from 0 to 8,589,934,592 in decimal. Clearly, the memory interface address range encompasses the packet address. From a code perspective, this should not produce an error.

To gain a deeper understanding of why this information results in an error, I examined the contains function to review its implementation. The specific implementation is as follows:

bool
    contains(const Addr& a) const
    {
        // check if the address is in the range and if there is either
        // no interleaving, or with interleaving also if the selected
        // bits from the address match the interleaving value
        bool in_range = a >= _start && a < _end;
        if (in_range) {
            auto sel = 0;
            for (unsigned int i = 0; i < masks.size(); i++) {
                Addr masked = a & masks[i];
                // The result of an xor operation is 1 if the number
                // of bits set is odd or 0 othersize, thefore it
                // suffices to count the number of bits set to
                // determine the i-th bit of sel.
                sel |= (popCount(masked) % 2) << i;
            }
            return sel == intlvMatch;
        }
        return false;
    }

The value of in_range is clearly 1, as indicated by the debugging information showing above that the packet address is encompassed by the memory interface address range. Therefore, the issue likely lies in the return value sel == intlvMatch.

So, according to the hints in this code snippet, it is likely either no interleaving, or with interleaving if the selected bits from the address match the interleaving value.

So, the core issue should be that the interleaving for HBMCtrl has not been implemented correctly.
So, I would like to know how to correctly implement interleaving for HBMCtrl.

Additionally, there is another issue. As is well known, HBM is rarely used as the sole memory in computer systems. Therefore, a memory system that includes HBM will usually also include DRAM. Is it possible to perform interleaving with both DRAM and HBM together in gem5? In my previous attempts, I only performed interleaving separately for DRAM and HBM. As a result, if HBM is located in a high address space, it may not be effectively accessible.

Thank you for any advice and assistance provided.

[LujhCoconut]

@mbabaie Thank you for your response to the issue I raised through email. Here is the specific and detailed information regarding the HBMCtrl issue.

[LujhCoconut]

I think I have partially resolved this issue. The problem indeed stems from incorrect settings during memory interleaving.
I referred to the article by the DArchR research group at UC Davis on simulating HBM2 controllers in gem5.
The article titled Modeling HBM2 Memory Controller was published in The 4th gem5 Users’ Workshop, held in conjunction with ISCA 2022.
Specific details about the article can be found on the homepage of the DArchR research group.
https://arch.cs.ucdavis.edu/memory/2022/06/01/hbm-gem5-workshop.html

Based on this article, I discovered that my issue was related to the granularity setting of memory interleaving.
The article points out that ”Our address mapping policy interleaves the memory requests across pseudo channels at a granularity of 64B (two 32B atoms for a 64B cache line go to the same pseudo channel).“

Therefore, when creating the memory configuration file, it is necessary to modify the corresponding intlv_size.
The correct value for intlv_size needs to be set to 64, whereas the default values are 128 or 512. This was something I overlooked. Thanks again to the DArchR research group for their contributions to gem5.

[LujhCoconut]

Through recent in-depth exploration of the gem5 source code, I have discovered a more refined answer to this issue.
In fact, gem5 already has code for implementing HBM2Stack in the file located at /src/python/gem5/components/memory/hbm.py. We just need to correctly understand this code to successfully configure HBMCtrl.

The code is quite short and easy to understand, so it will not be elaborated on here. Instead, a brief explanation of how to configure HBM2Stack will be provided.

Here, only the configuration code relevant to HBM2Stack is presented.

# Create address ranges
start = 0
end = 4*1024
mem_range = AddrRange(str(start) + 'MiB', str(end) + 'MiB')
system.mem_ranges.append(mem_range)

# Instantiate HBM2Stack and complete the correct configuration
 hbm_memory = HBM2Stack(size="4GiB")
 system.hbm_memory = hbm_memory
 system.hbm_memory._create_mem_interfaces_controller()
 system.hbm_memory.set_memory_range([system.mem_ranges[0]])
 system.hbm_memory._interleave_addresses()
 for ctrl in system.hbm_memory.mem_ctrl:
      ctrl.port = system.membus.mem_side_ports

[LujhCoconut]

One more thing , from gem5.components.memory import HBM2Stack