How to connect Ethernet device to ARM SMMU?

[sammy17]

I am trying to connect an ethernet device to ARM SMMU in a full system simulation.
Specifically, I am using the ARM dist setup where two simulations are connected through ethernet.
I want to connect this ethernet device to SMMU and SMMU will be connected to membus.

Documentation on SMMU is little to none and I only was able to find the comments given in the attachSmmu function.
According to this, I just need to attach the device using attachSmmu function and device should not be in _off_chip_devices and _on_chip_devices.

I tried using attachSmmu as below to connect the ethernet device in ARM dist_bigLITTLE.py example

gem5/configs/example/arm/dist_bigLITTLE.py

Line 138 in f29bfc0

def addEthernet(system, options):

addEthernet function before change:

def addEthernet(system, options):
    # create NIC
    dev = IGbE_e1000()
    system.attach_pci(dev)
    system.ethernet = dev

    # create distributed ethernet link

    system.etherlink = DistEtherLink(
        speed=options.ethernet_linkspeed,
        delay=options.ethernet_linkdelay,
        dist_rank=options.dist_rank,
        dist_size=options.dist_size,
        server_name=options.dist_server_name,
        server_port=options.dist_server_port,
        sync_start=options.dist_sync_start,
        sync_repeat=options.dist_sync_repeat,
    )
    system.etherlink.int0 = Parent.system.ethernet.interface
    if options.etherdump:
        system.etherdump = EtherDump(file=options.etherdump)
        system.etherlink.dump = system.etherdump

addEthernet function after change:

def addEthernet(system, options):
    # create NIC
    dev = IGbE_e1000()
    system.realview.attachSmmu([dev],system.membus)
    system.realview.smmu.irq_interface_enable = True
    system.ethernet = dev

    # create distributed ethernet link
    system.etherlink = DistEtherLink(
        speed=options.ethernet_linkspeed,
        delay=options.ethernet_linkdelay,
        dist_rank=options.dist_rank,
        dist_size=options.dist_size,
        server_name=options.dist_server_name,
        server_port=options.dist_server_port,
        sync_start=options.dist_sync_start,
        sync_repeat=options.dist_sync_repeat,
    )
    system.etherlink.int0 = Parent.system.ethernet.interface
    if options.etherdump:
        system.etherdump = EtherDump(file=options.etherdump)
        system.etherlink.dump = system.etherdump

I basically replaced system.attach_pci(dev) with system.realview.attachSmmu([dev],system.membus) and enabled IRQ interface in SMMU.
With this configuration, I get the below warning and error:

warn: system.realview.smmu adopting orphan SimObject param 'device_interfaces'
AttributeError: Can't resolve proxy 'any' of type 'PciHost' from 'system.ethernet'

As far as I understood, the cause for this error is that the host parameter of the ethernet device is not specified.
But in the case of attaching the device to SMMU, to what value should I specify the host parameter?
Is this the correct way to go about this?

[BobbyRBruce]

@andysan : Do you know anything about this?

[andysan]

The main problem here seems is that you always need to call system.attach_pci to allocate a PCI ID and wire the device to the PCI host controller which translates PCI subsystem interrupts to architecture-specific interrupts. Annoyingly, that call will also wire up the PIO and DMA ports which isn't necessarily what you want when using an SMMU.

There are two solutions you could try. The simplest solution would be to just add a call to system.realview.attachSmmu([dev],system.membus) after the system.attach_pci call in the original code. This should re-assign the NIC's DMA ports to the SMMU. There is a possibility that this doesn't work though because of idiosyncrasies in the way gem5 wires up buses.

The solution that should definitely work would be to do something along these lines:

def addEthernet(system, options):
    # Create an SMMU but don't connect it to any devices.
    system.realview.attachSmmu([], system.membus)
    # create NIC
    dev = IGbE_e1000()
    system.attach_pci(dev, dma_ports=[])
    system.ethernet = dev
    # Connect the device's DMA ports to the SMMU
    system.realview.smmu.connect(dev)

    ...

The code above first instantiates the SMMU without connecting it to any devices. This avoids wiring up the PIO port of the Ethernet device twice (there is a call to _attach_device in the attachSmmu helper and in attach_pci). The call to smmu.connect then creates an SMMU device interface that is connected to the device's DMA port.

[sammy17]

@giactra Sorry for the late reply. I am not providing the DTB manually to the simulation but I am not seeing the SMMU in the device tree that was generated as well. I instantiated SMMU in addEthernet (

gem5/configs/example/arm/dist_bigLITTLE.py

Line 138 in 48a40cf

def addEthernet(system, options):

) function as below: (according to the instructions given by andysan)

def addEthernet(system, options):
    # Create an SMMU but don't connect it to any devices.
    system.realview.attachSmmu([], system.membus)
    # create NIC
    dev = IGbE_e1000()
    system.attach_pci(dev, dma_ports=[])
    system.ethernet = dev
    # Connect the device's DMA ports to the SMMU
    system.realview.smmu.connect(dev)

I am not sure where to add self.addIommuProperty(state, node) as well. I understand that this should be called from the DMA device as below:
dev.addIommuProperty(state, node)
However, I am not sure where to get the state and node properties in this case.
Can you please give more information on where to add addIommuProperty and also let me know if I need to do anything extra to add SMMU to the DTB?

Also, one thing I noticed is in dist_bigLITTLE.py, addEthernet function is called after bL.build call. I noticed that DTB generation happens in build function. Is this the reason why I am not seeing SMMU in the DTB?

[sammy17]

I was able to generate the dt entry for SMMU by adding an additional call to root.system.generateDtb() after addEthernet function call in dist_bigLITTLE.py.
However, the issue with addIommuProperty remains.

Now Linux identifies the SMMU but no IOMMU groups are available (this makes sense as Linux does not know that the ethernet device is connected to the SMMU).
Where should I add addIommuProperty function call for the ethernet device in this case?

I tried to find the generateDeviceTree function for the ethernet device, but I did not see such a function for ethernet (or DMA devices in general).
Really appreciate any help on this.

[sammy17]

After further investigation, I saw that ethernet or any other PCI DMA device does not have a dedicated device tree node. All PCI DMA devices are connected to the PCI host and the PCI host has a single device tree node.

In this case, how can I mark that the PCI ethernet device is connected to the SMMU in the device tree?

After getting the SMMU in the device tree, I tried to boot Linux and set up network connection between the two simulations.
When I tried to set up the ethernet connection with ifconfig, the simulation crashed with the following error:

5354010558000: system.realview.smmu.device_interfaces: [a] req from system.ethernet.dma addr=0x81dd0000 size=0x40
5354010558000: system.realview.smmu.device_interfaces.port: IdReg:  p=0x7ffcf26c63b0 oid=9
5354010558000: system.realview.smmu.device_interfaces.port: micro TLB miss vaddr=0x81dd0000 sid=0 ssid=0
5354010558000: system.realview.smmu.device_interfaces.port: RESPONSE Interface TLB miss vaddr=0x81dd0000 sid=0 ssid=0
5354010558000: system.realview.smmu.device_interfaces.port: 4kReg:  p=0x7ffcf26c63b0 a4k=0x81dd0000
5354010558000: system.realview.smmu.device_interfaces.port: Config miss sid=0 ssid=0
5354010558000: system.realview.smmu.device_interfaces.port: Read STE at 0x81abd000
5354010558000: system.realview.smmu.device_interfaces.port: Got STE at 0x81abd000 [0]: 0x0000000000000001
5354010558000: system.realview.smmu.device_interfaces.port:     STE at 0x81abd000 [1]: 0x0000100000000000
5354010558000: system.realview.smmu.device_interfaces.port:     STE at 0x81abd000 [2]: 0x0000000000000000
5354010558000: system.realview.smmu.device_interfaces.port:     STE at 0x81abd000 [3]: 0x0000000000000000
5354010558000: system.realview.smmu.device_interfaces.port:     STE at 0x81abd000 [4]: 0x0000000000000000
5354010558000: system.realview.smmu.device_interfaces.port:     STE at 0x81abd000 [5]: 0x0000000000000000
5354010558000: system.realview.smmu.device_interfaces.port:     STE at 0x81abd000 [6]: 0x0000000000000000
5354010558000: system.realview.smmu.device_interfaces.port:     STE at 0x81abd000 [7]: 0x0000000000000000
src/dev/arm/smmu_v3_transl.cc:612: panic: Bad or unimplemented STE config 0

Please let me know if you have any suggestions for resolving this issue @andysan @giactra

[andysan]

There should be no need to include the PCI Ethernet device in the device tree since PCI is a discoverable bus. The SMMU in a real system would be connected to the PCI root complex and use stream IDs to distinguish different PCI cards. I.e., the SMMU needs to be associated with the PCI root complex. I don't think we automatically generate the right DT fragments to do that automatically at the moment. The relevant kernel documentation can be found here:

• https://elixir.bootlin.com/linux/v6.6-rc3/source/Documentation/devicetree/bindings/pci/pci-iommu.txt
• https://elixir.bootlin.com/linux/v6.6-rc3/source/Documentation/devicetree/bindings/iommu/iommu.txt
• https://elixir.bootlin.com/linux/v6.6-rc3/source/Documentation/devicetree/bindings/iommu/arm,smmu-v3.yaml

[sammy17]

Thank you very much @andysan !
I finally got it working by manually editing the pci_host device tree node as you mentioned.

	pci {
		compatible = "pci-host-ecam-generic";
		device_type = "pci";
		#address-cells = <0x03>;
		#size-cells = <0x02>;
		#interrupt-cells = <0x01>;
		reg = <0x00 0x40000000 0x00 0x10000000>;
		ranges = <0x1000000 0x00 0x00 0x00 0x50000000 0x00 0x10000 0x2000000 0x00 0x00 0x40 0x00 0x00 0x40000000>;
		interrupt-map = <0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x44 0x01 0x800 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x45 0x01 0x1000 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x46 0x01 0x1800 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x47 0x01>;
		interrupt-map-mask = <0x1800 0x00 0x00 0x00>;
		dma-coherent;
		iommu-map = <0x10 &iommu 0x0 0x1>;
	};

I added the iommu-map entry for only translating the requests of the ethernet device.