Implementing Multi-GPU Support in gem5: Modifying mmap to Include GPU ID

[cja1229]

Hello ,
I'm currently working on adding multi-GPU support in gem5, and I've run into a challenge. I need to modify the mmap function in src/gpu-compute/gpu-compute-driver.cc to include the GPU ID.
Could anyone provide guidance on how to implement this? Specifically, I’d appreciate any insights on what changes are necessary to ensure that mmap properly handles GPU IDs for multiple instances.
Thank you for your help!

GPUComputeDriver::mmap(ThreadContext *tc, Addr start, uint64_t length,
                       int prot, int tgt_flags, int tgt_fd, off_t offset)
{
    auto process = tc->getProcessPtr();
    auto mem_state = process->memState;

    Addr pg_off = offset >> PAGE_SHIFT;
    Addr mmap_type = pg_off & KFD_MMAP_TYPE_MASK;
    DPRINTF(GPUDriver, "amdkfd mmap (start: %p, length: 0x%x,"
            "offset: 0x%x)\n", start, length, offset);

    switch(mmap_type) {
        case KFD_MMAP_TYPE_DOORBELL:
            DPRINTF(GPUDriver, "amdkfd mmap type DOORBELL offset\n");
            start = mem_state->extendMmap(length);
            process->pTable->map(start, device->hsaPacketProc().pioAddr,
                    length, false);
            break;
        case KFD_MMAP_TYPE_EVENTS:
            DPRINTF(GPUDriver, "amdkfd mmap type EVENTS offset\n");
            panic_if(start != 0,
                     "Start address should be provided by KFD\n");
            panic_if(length != 8 * KFD_SIGNAL_EVENT_LIMIT,
                     "Requested length %d, expected length %d; length "
                     "mismatch\n", length, 8* KFD_SIGNAL_EVENT_LIMIT);
            /**
             * We don't actually access these pages.  We just need to reserve
             * some VA space.  See commit id 5ce8abce for details on how
             * events are currently implemented.
             */
            if (!eventPage) {
                eventPage = mem_state->extendMmap(length);
                start = eventPage;
            }
            break;
        default:
            warn_once("Unrecognized kfd mmap type %llx\n", mmap_type);
            break;
    }

    return start;
}

[mattsinc]

I'd need to know more about the context here (e.g., why you need GPU ID here) to provide help. However, my group has recently released our multi-chiplet GPU support (although there is a lot to improve on it) as part of our upcoming MICRO '24 paper. It still needs to be integrated into the mainline too, so your work may still be valuable. If you shoot me an email I can explain where that support is at, and we can discuss if what you are doing is better than trying to leverage our stuff.

[cja1229]

Thank you for your reply! I found this link : gem5 Multi-GPU Guide, which explains how a unique GPU ID parameter is added to distinguish each component. However, I’m still a bit confused about how this works in practice.
In the code, particularly in src/gpu-compute/gpu-compute-driver.cc in gem5, it seems to be represented by dGPUPoolID. I appreciate any further insights you might have!

[mattsinc]

Yes, that is an article I/we wrote several years ago. Certainly each chiplet/GPU has a unique ID. The mainline gem5 code does not have any of that code integrated though, so I would have to look at what dGPUPoolID represents -- it would be a surprise if it was related to this, but I could be wrong.

Regardless as I mentioned above we are working on releasing that support you linked as part of our MICRO paper. That public support is not completely working yet for reasons I'm still working through, but if your goal is to just get that code you could work with us to verify what is released publicly is useful. There are downsides though -- for example, that version of the code assumes ROCm 1.6 support, which is not the latest and greatest ROCm version anymore. And it is on our todo list to get chiplet support for more modern ROCm versions.

This sort of discussion is probably best done via email though.

[cja1229]

Thank you for your response!I’ve sent an email regarding my questions about multi-GPU support and the related code. I would greatly appreciate it if you could reply to me.