OpenOnload not able to allocate stacks.

[shirshen12]

I have trying to get OpenOnload to work on a bare-metal box from Vultr.io

Environment Details:
CPU: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
NIC: Intel Corporation 82599 10 Gigabit Network Connection
OS: CentOS Linux release 8.2.2004 (Core)
Kernel: 4.18.0-193.28.1.el8_2.x86_64

How to Build and Install Onload:
wget https://rpmfind.net/linux/centos/8.2.2004/BaseOS/x86_64/os/Packages/kernel-headers-4.18.0-193.28.1.el8_2.x86_64.rpm
wget https://rpmfind.net/linux/centos/8.2.2004/BaseOS/x86_64/os/Packages/kernel-devel-4.18.0-193.28.1.el8_2.x86_64.rpm
yum install binutils gettext gawk gcc sed make bash glibc-common libcap-devel libmnl-devel perl-Test-Harness hmaccalc zlib-devel binutils-devel elfutils-libelf-devel libevent-devel
git clone https://github.com/Xilinx-CNS/onload.git
cd onload
scripts/onload_mkdist --release
cd onload-20201211/scripts
./onload_install
./onload_tool reload

Verify Onload driver has been built/inserted

[root@shirbare06 scripts]# lsmod | grep sfc sfc_char 98304 1 onload sfc_resource 180224 2 onload,sfc_char sfc 581632 0 virtual_bus 16384 1 sfc sfc_driverlink 16384 2 sfc,sfc_resource mtd 69632 1 sfc mdio 16384 2 sfc,ixgbe

**Check if XDP module has been enabled for via xdptools **
Install XDP tools
yum install clang llvm
dnf --enablerepo=PowerTools install libpcap-devel
git clone https://github.com/xdp-project/xdp-tools.git
cd xdp-tools
./configure
make
sudo make install

Check if xdpsock() is enabled:
`[root@shirbare06 scripts]# xdpdump -D
if_index if_name XDP program entry function

1 lo <No XDP program loaded!>
2 enp1s0 xdpsock()`

Start a sample memcached
s.chakrabarti@shirbare06 ~]$ onload -p latency memcached -m 24576 -c 1024 -t 8 -u s.chakrabarti -l 95.179.137.242:11211 oo:memcached[96975]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1) See kernel messages in dmesg or /var/log/syslog for more details of this failure

Also verified stack is not being created:
[root@shirbare06 scripts]# onload_stackdump stacks

Can someone please look at this issue and fix :)

[sianj-xilinx]

Firstly, thanks for trying this out! Our AF_XDP support is a work in progress, so it's great to have someone else having a go.

It doesn't look like you're running as root, which you need to be to create the stack, as this is needed for setting up the AF_XDP resources.

If that doesn't resolve the problem, then could you please attach your dmesg output? We try to log a bit more information about what's going on there.

[shirshen12]

Hello @sianj-xilinx ,

I did try running as root, as recommended by you but still no luck.

[root@shirbare07 ~]# onload -p latency memcached -m 24576 -c 1024 -t 8 -u s.chakrabarti -l 45.32.238.185:11211
oo:memcached[70641]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1) See kernel messages in dmesg or /var/log/syslog for more details of this failure

Output of onload_stackdump:
[root@shirbare07 ~]# onload_stackdump stacks
[root@shirbare07 ~]#

dmesg output:

[shirshen12]

Hello @sianj-xilinx ,

I think you edited your README that its a WIP and you are right. The dmesg error logs do point to the following functions not being implemented:

af_xdp_flush_tx_dma_channel()
af_xdp_flush_rx_dma_channel()

There are link 15 such functions in src/lib/efhw/af_xdp.c

Request you to take a look at them.

[maciejj-xilinx]

Hello @shirshen12,
From our tests ixgbe requires prior enabling of flow director:
ethtool --features eth0 ntuple on

Let us know if this resolves the issue for you.

[shirshen12]

Hello @maciejj-xilinx ,

I did turn on the flow director for 82599 ixgbe driver

[root@shirbare08 ~]# ethtool --show-features enp1s0 | grep ntuple
ntuple-filters: on

I compiled the latest onload driver:

I see the same error as above:
[root@shirbare08 ~]# onload -p latency memcached -m 24576 -c 1024 -t 8 -u s.chakrabarti -l 136.244.97.136:11211
oo:memcached[92470]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1)
See kernel messages in dmesg or /var/log/syslog for more details of this failure

Also, confirmed via onload_stackdump stacks
[root@shirbare08 onload]# onload_stackdump stacks
[root@shirbare08 onload]#

I am seeing the issue in dmesg being logged:

[shirshen12]

Also @maciejj-xilinx / @sianj-xilinx can you share your env details under which you are testing. Since, if its going through for you I can replicate your environment details.

[maciejj-xilinx]

I have just checked memcached command line options. It looks -u s.chakrabarti instructs memcached to drop priviledges.
This happens before listen socket (and thus onload stack) is created.
I'd suggest trying -u root as a workaround.

Nonetheless, this is genuine issue, and we can see it needs to resolved - that is to allow creating stacks by unpriviledged users.

[shirshen12]

Ok let me try that and revert.

[shirshen12]

Hello @maciejj-xilinx ,

It worked, please see below for confirmation.

[root@shirbare09 ~]# onload -p latency memcached -m 24576 -c 1024 -t 8 -u root -l 136.244.100.84:11211
oo:memcached[69454]: Using Onload 20201215 [1]
oo:memcached[69454]: Copyright 2019-2020 Xilinx, 2006-2019 Solarflare Communications, 2002-2005
Level 5 Networks

Stacks being created:
[root@shirbare09 ~]# onload_stackdump
#stack-id stack-name pids
1 - 69454

I will close this ticket and I will raise a new ticket for non privileged users should be able to create stacks.

I will test on some other NiCs like MLNX ConnectX and report on AF_XDP stuff as well. For Intel 82599, it works as of now.