make run-all-tests fails (sending with wrong VIP and verifying drop FAILS)

[mhanif]

Just cloned a fresh copy of DASH at the tip of master branch. Built and tested (unmodified) to get the baseline and saw that it is failing:

$ make run-saithrift-server
docker run --rm -it --net=host --name dash-saithrift-server-mhanif -v /home/mhanif/sw/dash-ipv6/DASH/dash-pipeline/bmv2/dash_pipeline.bmv2/dash_pipeline.json:/etc/dash/dash_pipeline.json -v /home/mhanif/sw/dash-ipv6/DASH/dash-pipeline/bmv2/dash_pipeline.bmv2/dash_pipeline_p4rt.txt:/etc/dash/dash_pipeline_p4rt.txt -v /home/mhanif/sw/dash-ipv6/DASH/dash-pipeline/SAI:/SAI -v /home/mhanif/sw/dash-ipv6/DASH/dash-pipeline/SAI/SAI/meta:/meta -w /SAI/rpc/usr/sbin -e LD_LIBRARY_PATH=/SAI/lib:/usr/local/lib chrissommers/dash-saithrift-bldr:220719
./saiserver
GRPC call SetForwardingPipelineConfig 0.0.0.0:9559 => /etc/dash/dash_pipeline.json, /etc/dash/dash_pipeline_p4rt.txt
Starting SAI RPC server on port 9092
GRPC call Write::INSERT OKtable_id: 45621839 match { field_id: 1 exact { value: "\254\020\001d" } } action { action { action_id: 26041632 } }
GRPC call Write::INSERT OKtable_id: 41103078 match { field_id: 1 exact { value: "\000\000d" } } action { action { action_id: 21912829 } }
GRPC call Write::INSERT OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\001" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::INSERT OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\002" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::INSERT OKtable_id: 34579306 match { field_id: 1 exact { value: "\000\003" } } action { action { action_id: 17187022 params { param_id: 1 value: "\000\000d" } } }
GRPC call Write::INSERT OKtable_id: 37987821 match { field_id: 1 exact { value: "\000\004" } } action { action { action_id: 27167550 params { param_id: 1 value: "\000\000'\020" } params { param_id: 2 value: "\000\001\206\240" } params { param_id: 3 value: "\000\001\206\240" } params { param_id: 4 value: "\001" } params { param_id: 5 value: "\254\020\001\001" } params { param_id: 6 value: "\000\000\t" } params { param_id: 7 value: "\000\003" } params { param_id: 8 value: "\000\000" } params { param_id: 9 value: "\000\000" } params { param_id: 10 value: "\000\000" } params { param_id: 11 value: "\000\000" } params { param_id: 12 value: "\000\000" } params { param_id: 13 value: "\000\000" } params { param_id: 14 value: "\000\000" } params { param_id: 15 value: "\000\000" } params { param_id: 16 value: "\000\000" } params { param_id: 17 value: "\000\000" } params { param_id: 18 value: "\000\000" } params { param_id: 19 value: "\000\000" } params { param_id: 20 value: "\000\000" } params { param_id: 21 value: "\000\000" } params { param_id: 22 value: "\000\000" } params { param_id: 23 value: "\000\000" } params { param_id: 24 value: "\000\000" } params { param_id: 25 value: "\000\000" } params { param_id: 26 value: "\000\000" } params { param_id: 27 value: "\000\000" } } }
GRPC call Write::INSERT OKtable_id: 37820776 match { field_id: 1 exact { value: "\000\314\314\314\314\314" } } action { action { action_id: 18228884 params { param_id: 1 value: "\000\004" } } }
GRPC call Write::INSERT OKtable_id: 37768533 match { field_id: 1 exact { value: "\000\004" } } match { field_id: 3 lpm { value: "\n\001\000\000" prefix_len: 112 } } match { field_id: 2 exact { value: "\000" } } action { action { action_id: 25364446 params { param_id: 1 value: "\000\003" } } }
GRPC call Write::INSERT OKtable_id: 34260339 match { field_id: 1 exact { value: "\000\003" } } match { field_id: 3 exact { value: "\n\001\0022" } } match { field_id: 2 exact { value: "\000" } } action { action { action_id: 25450412 params { param_id: 1 value: "\254\020\001\024" } params { param_id: 2 value: "\000\335\335\335\335\335" } params { param_id: 3 value: "\001" } } }
GRPC call Write::DELETE OKtable_id: 34260339 match { field_id: 1 exact { value: "\000\003" } } match { field_id: 3 exact { value: "\n\001\0022" } } match { field_id: 2 exact { value: "\000" } }
GRPC call Write::DELETE OKtable_id: 37768533 match { field_id: 1 exact { value: "\000\004" } } match { field_id: 3 lpm { value: "\n\001\000\000" prefix_len: 112 } } match { field_id: 2 exact { value: "\000" } }
GRPC call Write::DELETE OKtable_id: 37820776 match { field_id: 1 exact { value: "\000\314\314\314\314\314" } }
GRPC call Write::DELETE OKtable_id: 37987821 match { field_id: 1 exact { value: "\000\004" } } action { action { action_id: 27167550 params { param_id: 1 value: "\000\000'\020" } params { param_id: 2 value: "\000\001\206\240" } params { param_id: 3 value: "\000\001\206\240" } params { param_id: 4 value: "\001" } params { param_id: 5 value: "\254\020\001\001" } params { param_id: 6 value: "\000\000\t" } params { param_id: 7 value: "\000\003" } params { param_id: 8 value: "\000\000" } params { param_id: 9 value: "\000\000" } params { param_id: 10 value: "\000\000" } params { param_id: 11 value: "\000\000" } params { param_id: 12 value: "\000\000" } params { param_id: 13 value: "\000\000" } params { param_id: 14 value: "\000\000" } params { param_id: 15 value: "\000\000" } params { param_id: 16 value: "\000\000" } params { param_id: 17 value: "\000\000" } params { param_id: 18 value: "\000\000" } params { param_id: 19 value: "\000\000" } params { param_id: 20 value: "\000\000" } params { param_id: 21 value: "\000\000" } params { param_id: 22 value: "\000\000" } params { param_id: 23 value: "\000\000" } params { param_id: 24 value: "\000\000" } params { param_id: 25 value: "\000\000" } params { param_id: 26 value: "\000\000" } params { param_id: 27 value: "\000\000" } } }
GRPC call Write::DELETE OKtable_id: 34579306 match { field_id: 1 exact { value: "\000\003" } } action { action { action_id: 17187022 params { param_id: 1 value: "\000\000d" } } }
GRPC call Write::DELETE OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\002" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::DELETE OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\001" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::DELETE OKtable_id: 41103078 match { field_id: 1 exact { value: "\000\000d" } }
GRPC call Write::DELETE OKtable_id: 45621839 match { field_id: 1 exact { value: "\254\020\001d" } }

---

$ make run-all-tests

Ensure P4Runtime server is listening

t=5;
while [ ${t} -ge 1 ]; do
if sudo lsof -i:9559 | grep LISTEN >/dev/null; then
break;
else
sleep 1;
t=expr $t - 1;
fi;
done;
docker exec -w /tests/libsai/vnet_out simple_switch-mhanif ./vnet_out
[sudo] password for mhanif:
GRPC call SetForwardingPipelineConfig 0.0.0.0:9559 => /etc/dash/dash_pipeline.json, /etc/dash/dash_pipeline_p4rt.txt
GRPC call Write::INSERT OKtable_id: 41103078 match { field_id: 1 exact { value: "\000\000<" } } action { action { action_id: 21912829 } }
GRPC call Write::INSERT OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\001" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::INSERT OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\002" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::INSERT OKtable_id: 34579306 match { field_id: 1 exact { value: "\000\003" } } action { action { action_id: 17187022 params { param_id: 1 value: "\000\000\t" } } }
GRPC call Write::INSERT OKtable_id: 37987821 match { field_id: 1 exact { value: "\000\004" } } action { action { action_id: 27167550 params { param_id: 1 value: "\000\000'\020" } params { param_id: 2 value: "\000\001\206\240" } params { param_id: 3 value: "\000\001\206\240" } params { param_id: 4 value: "\001" } params { param_id: 5 value: "\254\020\003\001" } params { param_id: 6 value: "\000\000\t" } params { param_id: 7 value: "\000\003" } params { param_id: 27 value: "\000\000" } params { param_id: 26 value: "\000\000" } params { param_id: 25 value: "\000\000" } params { param_id: 24 value: "\000\000" } params { param_id: 23 value: "\000\000" } params { param_id: 17 value: "\000\000" } params { param_id: 16 value: "\000\000" } params { param_id: 15 value: "\000\000" } params { param_id: 14 value: "\000\000" } params { param_id: 13 value: "\000\000" } params { param_id: 22 value: "\000\002" } params { param_id: 21 value: "\000\002" } params { param_id: 20 value: "\000\002" } params { param_id: 19 value: "\000\002" } params { param_id: 18 value: "\000\002" } params { param_id: 12 value: "\000\001" } params { param_id: 11 value: "\000\001" } params { param_id: 10 value: "\000\001" } params { param_id: 9 value: "\000\001" } params { param_id: 8 value: "\000\001" } } }
GRPC call Write::INSERT OKtable_id: 37820776 match { field_id: 1 exact { value: "\252\314\314\314\314\314" } } action { action { action_id: 18228884 params { param_id: 1 value: "\000\004" } } }
GRPC call Write::DELETE OKtable_id: 37820776 match { field_id: 1 exact { value: "\252\314\314\314\314\314" } }
GRPC call Write::DELETE OKtable_id: 37987821 match { field_id: 1 exact { value: "\000\004" } } action { action { action_id: 27167550 params { param_id: 1 value: "\000\000'\020" } params { param_id: 2 value: "\000\001\206\240" } params { param_id: 3 value: "\000\001\206\240" } params { param_id: 4 value: "\001" } params { param_id: 5 value: "\254\020\003\001" } params { param_id: 6 value: "\000\000\t" } params { param_id: 7 value: "\000\003" } params { param_id: 27 value: "\000\000" } params { param_id: 26 value: "\000\000" } params { param_id: 25 value: "\000\000" } params { param_id: 24 value: "\000\000" } params { param_id: 23 value: "\000\000" } params { param_id: 17 value: "\000\000" } params { param_id: 16 value: "\000\000" } params { param_id: 15 value: "\000\000" } params { param_id: 14 value: "\000\000" } params { param_id: 13 value: "\000\000" } params { param_id: 22 value: "\000\002" } params { param_id: 21 value: "\000\002" } params { param_id: 20 value: "\000\002" } params { param_id: 19 value: "\000\002" } params { param_id: 18 value: "\000\002" } params { param_id: 12 value: "\000\001" } params { param_id: 11 value: "\000\001" } params { param_id: 10 value: "\000\001" } params { param_id: 9 value: "\000\001" } params { param_id: 8 value: "\000\001" } } }
GRPC call Write::DELETE OKtable_id: 34579306 match { field_id: 1 exact { value: "\000\003" } } action { action { action_id: 17187022 params { param_id: 1 value: "\000\000\t" } } }
Done.
GRPC call Write::DELETE OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\002" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::DELETE OKtable_id: 45323240 match { field_id: 1 exact { value: "\000\001" } } action { action { action_id: 25655048 params { param_id: 1 value: "\000\000\000\000" } } }
GRPC call Write::DELETE OKtable_id: 41103078 match { field_id: 1 exact { value: "\000\000<" } }
cd ../test/third-party/traffic_gen && ./deploy_ixiac.sh
.
Creating ixia-c-traffic-engine1-mhanif ... done
Creating ixia-c-traffic-engine2-mhanif ... done
Creating ixia-c-controller-mhanif ... done
docker run -u root --network=host --rm -w /tests/ -it
-w /tests/saithrift/ptf
local/dash-saithrift-client:latest
./run-saithrift-ptftests.sh
/usr/local/lib/python3.8/dist-packages/ptf-0.9.3-py3.8.egg/EGG-INFO/scripts/ptf:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
Using packet manipulation module: ptf.packet_scapy
test_thrift_session.TestSaiThriftSession ... TestSaiThriftSession OK
ok

---

Ran 1 test in 0.002s

OK
test_thrift_session.TestSaiThriftSaiHelper ... TestSaiThriftSaiHelper OK
ok

---

Ran 1 test in 0.001s

OK
test_saithrift_vnet.TestSaiThrift_outbound_udp_pkt ...

Sending packet with wrong vip...

<Ether dst=00:00:02:03:04:05 src=00:00:05:06:06:06 type=IPv4 |<IP ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=172.16.1.1 dst=172.16.100.100 |<UDP sport=11638 dport=4789 chksum=0x0 |<VXLAN flags=Instance reserved1=0 vni=0x64 reserved2=0x0 |<Ether dst=02:02:02:02:02:02 src=00:cc:cc:cc:cc:cc type=IPv4 |<IP ihl=None tos=0x0 id=1 flags= frag=0 ttl=64 proto=udp src=10.1.1.10 dst=10.1.2.50 |<UDP sport=1234 dport=80 |<Raw load='\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&'()*+,-./0123456789' |>>>>>>>>

Verifying drop...
FAIL

======================================================================
FAIL: test_saithrift_vnet.TestSaiThrift_outbound_udp_pkt

Traceback (most recent call last):
File "./vnet/test_saithrift_vnet.py", line 225, in runTest
raise ae
File "./vnet/test_saithrift_vnet.py", line 146, in runTest
verify_no_other_packets(self)
File "/usr/local/lib/python3.8/dist-packages/ptf-0.9.3-py3.8.egg/ptf/testutils.py", line 3291, in verify_no_other_packets
test.fail(
AssertionError: A packet was received on device 0, port 0, but we expected no packets.
========== RECEIVED ==========
0000 33 33 00 00 00 16 7A EF 5F 17 69 A9 86 DD 60 00 33....z._.i...`.
0010 00 00 00 4C 00 01 FE 80 00 00 00 00 00 00 78 EF ...L..........x.
0020 5F FF FE 17 69 A9 FF 02 00 00 00 00 00 00 00 00 _...i...........
0030 00 00 00 00 00 16 3A 00 05 02 00 00 01 00 8F 00 ......:.........
0040 C1 64 00 00 00 03 04 00 00 00 FF 02 00 00 00 00 .d..............
0050 00 00 00 00 00 01 FF 00 00 00 04 00 00 00 FF 02 ................
0060 00 00 00 00 00 00 00 00 00 01 FF 17 69 A9 04 00 ............i...
0070 00 00 FF 02 00 00 00 00 00 00 00 00 00 00 00 00 ................
0080 00 02 ..

---

Ran 1 test in 0.138s

FAILED (failures=1)

---

ATTENTION: SOME TESTS DID NOT PASS!!!

The following tests failed:
TestSaiThrift_outbound_udp_pkt

---

make: *** [Makefile:374: run-saithrift-ptftests] Error 1

---

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
35725a3a4693 ixiacom/ixia-c-traffic-engine:1.4.1.26 "./entrypoint.sh" 4 minutes ago Up 4 minutes ixia-c-traffic-engine2-mhanif
584cfe0175de ixiacom/ixia-c-controller:0.0.1-2934 "./bin/controller --…" 4 minutes ago Up 4 minutes ixia-c-controller-mhanif
478016d7cb2f ixiacom/ixia-c-traffic-engine:1.4.1.26 "./entrypoint.sh" 4 minutes ago Up 4 minutes ixia-c-traffic-engine1-mhanif
bbf3233c21f5 chrissommers/dash-saithrift-bldr:220719 "./saiserver" 8 minutes ago Up 8 minutes dash-saithrift-server-mhanif
509dc995d6ed chrissommers/dash-bmv2-bldr:220630 "env LD_LIBRARY_PATH…" About an hour ago Up About an hour simple_switch-mhanif

[chrispsommers]

@mhanif Thanks for reporting. I saw something similar while I was working the other day, but it somehow resolved and I can't say exactly how, but it might have to do with version of PTF and p4c. I was developing/testing #202. You might want to clone my dev fork https://github.com/chrispsommers/, checkout branch fix-docker-perms and give that a try.

The curious thing is this version appears to run CI tests fine, so it is possible the failure results from some previous state. Did you make docker-saithrift-client ?

[mhanif]

The curious thing is this version appears to run CI tests fine, so it is possible the failure results from some previous state. Did you make docker-saithrift-client ?

@chrispsommers I did "make all" which does do " make docker-saithrift-client". I will try with your "fix-docker-perms" branch and see if I have any luck :-) Thanks!

[mukeshmv]

This looks like a stray MLD report ICMPv6 packet. So we may need to guard against that - this is something Chris had commented earlier when the drop packet test was introduced.

Internet Protocol Version 6
    0110 .... = Version: 6
    .... 0000 0000 .... .... .... .... .... = Traffic Class: 0x00 (DSCP: CS0, ECN: Not-ECT)
    .... 0000 0000 0000 0000 0000 = Flow Label: 0x00000
    Payload Length: 76
    Next Header: IPv6 Hop-by-Hop Option (0)
    Hop Limit: 1
    Source Address: fe80::78ef:5fff:fe17:69a9
    Destination Address: ff02::16
    IPv6 Hop-by-Hop Option
Internet Control Message Protocol v6
    Type: Multicast Listener Report Message v2 (143)
    Code: 0
    Checksum: 0xc164 [correct]
    Checksum Status: Good
    Reserved: 0000
    Number of Multicast Address Records: 3
    Multicast Address Record Changed to exclude: ff02::1:ff00:0
    Multicast Address Record Changed to exclude: ff02::1:ff17:69a9
    Multicast Address Record Changed to exclude: ff02::2

[mukeshmv]

Our Makefile has these lines to disable IPv6

network:
  sudo sysctl net.ipv6.conf.veth0.disable_ipv6=1
  sudo sysctl net.ipv6.conf.veth1.disable_ipv6=1
  sudo sysctl net.ipv6.conf.veth2.disable_ipv6=1
  sudo sysctl net.ipv6.conf.veth3.disable_ipv6=1

And when I check in my system I see that ipv6 is disabled on these interfaces.

mukesh@mukesh$ sysctl net.ipv6.conf.veth0.disable_ipv6
net.ipv6.conf.veth0.disable_ipv6 = 1
mukesh@mukesh$ sysctl net.ipv6.conf.veth1.disable_ipv6
net.ipv6.conf.veth1.disable_ipv6 = 1
mukesh@mukesh$ sysctl net.ipv6.conf.veth2.disable_ipv6
net.ipv6.conf.veth2.disable_ipv6 = 1
mukesh@mukesh$ sysctl net.ipv6.conf.veth3.disable_ipv6
net.ipv6.conf.veth3.disable_ipv6 = 1

@mhanif can you check this on your side. With this I believe we should not be seeing IPv6 packets.

[chrispsommers]

@mukeshmv thanks for the insight, I missed that. It might explain the sporadic nature of this problem. Despite all my efforts to prevent IPv6 contamination of the test-cases (all those sysctl commands I added to the initial ones provided by @marian-pritsak), it still seems to creep in - you can see in Wireshark monitoring veth's. I've previously spent hours on this and didn't get a perfect solution. Maybe a few more magic sysclt commands would solve it. Perhaps the use of container and device namespaces would help. Any advice is welcome.

[mukeshmv]

@mhanif if the disable_ipv6 sysctl parameters are already set in your case and you are still seeing the issue can you try setting these MLD params to see if it helps

sudo sysctl -w net.ipv6.conf.veth0.mldv2_unsolicited_report_interval=0
sudo sysctl -w net.ipv6.conf.veth1.mldv2_unsolicited_report_interval=0
sudo sysctl -w net.ipv6.conf.veth2.mldv2_unsolicited_report_interval=0
sudo sysctl -w net.ipv6.conf.veth3.mldv2_unsolicited_report_interval=0

Or if 0 is not switching it off then alternatively you could try increasing the interval to a very high value (set in millisecs).

By the way, I don't see any IPv6 packet in my envrironment on these 4 interfaces even before making any changes. I tried capturing using tcpdump before, after and while I was running the tests.

[chrispsommers]

I have some more empirical evidence. In my environment, I see numerous IPv6 packets appear on the veths after executing the following sequence. You don't need to make run-switch or make run-saithrift-server.

• make network # console 1 - creates veths
• sudo tcpdump -i veth0 # console 2
• make deploy-ixiac # console 1
  dash-pkt-capture.txt

You can see IPv6 packets appear on the veths over a period of a few seconds. See attachment for example. This seems to happen only on first invocation of make network and make deploy-ixiac and test failure seems to be environment-dependent.

I will investigate further why this happens. In the meantime, I think the remedy could be to delete the deploy-ixiac prerequisite from the following make target located here: https://github.com/Azure/DASH/blob/191bb85bf07c7746ef81113e590cdb04cb7e12e8/dash-pipeline/Makefile#L6

The apparent impact of this prerequisite is to produce these IPv6 packets during the execution of the subsequent run-saithrift-ptftests (one of the components of run-saithrift-client-tests) which breaks the test because "expected packet" ends up being a random IPv6 packet. Removing the prerequisite won't interfere with PTF tests because ixia-c isn't used there. (In fact there are no tests currently enabled using ixia-c.) In any case, deploy-ixia-c is also a prerequisite for run-saithrift-pytests so it's OK to remove from run-all-tests.

Sorry for the long explanation. In summary:

• @mhanif could you please try removing deploy-ixiac from run-all-tests and retry the usual make run-switch, make run-saithrift-server, make run-all-tests? Please execute make network-clean undeploy-ixiac first to get to the initial conditions.
• Assuming this works, I can do a PR to remedy this in the near-term
• I will pursue the root cause of the IPv6 packets emitted during deploy-ixiac

[mukeshmv]

Good find @chrispsommers. I was only doing run-saithrift-ptftests after make all .. thats why I didn't see it.

And I also noticed that IPv6 is getting enabled back after make deploy-ixiac.

sysctl net.ipv6.conf.veth0.disable_ipv6
net.ipv6.conf.veth0.disable_ipv6 = 0

So I guess we don't need to dig further into sysctls. Just keeping the current one disabled should work.

[mhanif]

@chrispsommers @mukeshmv Thanks a lot for all your help and suggestions. After todays meeting, I re-ran the command "make run-all-tests" and it passed with the message - "3 passed, 1 skipped in 0.76s". Not sure if it is supposed to skip one test? In any case, I didn't make any changes to my environment or issued any sysctl command. I will re-run this a few times to see if there are intermittent failures. In case of any failures, I will try your suggestions. Thanks again.

[chrispsommers]

@mhanif Thanks and you're welcome!

Indeed, one test was commented out awhile back so it is "skipped." It's here:
https://github.com/Azure/DASH/blob/66671be26f56c431208241046ce96d22d022e666/dash-pipeline/tests/saithrift/pytest/echo/test_echo_port.py#L6

@mukesh, can this be reenabled yet? I forget the rationale here.

[mhanif]

@chrispsommers After I re-ran the tests few more time and I occasionally (more than once) see that the test stops right away with the following message. Haven't debug to see why

$ make run-all-tests

Ensure P4Runtime server is listening

t=5;
while [ ${t} -ge 1 ]; do
if sudo lsof -i:9559 | grep LISTEN >/dev/null; then
break;
else
sleep 1;
t=expr $t - 1;
fi;
done;
docker exec -w /tests/libsai/vnet_out simple_switch-mhanif ./vnet_out
GRPC call SetForwardingPipelineConfig 0.0.0.0:9559 => /etc/dash/dash_pipeline.json, /etc/dash/dash_pipeline_p4rt.txt
GRPC ERROR[7]: Not primary, GRPC call Write::INSERT ERROR:
table_id: 41103078 match { field_id: 1 exact { value: "\000\000<" } } action { action { action_id: 21912829 } }Failed to create Direction Lookup Entry
make: *** [Makefile:253: run-libsai-test] Error 1

[chrispsommers]

Hi @mhanif It seems like the P4Runtime server inside bmv2 is getting a client connection request when another P4Runtime client already has "mastership." The clue is: Not primary. Check to see which containers are running via docker ps in case something is leftover.

[mukeshmv]

@mukesh, can this be reenabled yet? I forget the rationale here.

@chrispsommers no dash pipeline does not support echo of non-vxlan packet. If I remember right we concluded that you will go ahead and replicate the PTF outbound packet test as a Pytest. That skip reason given is outdated, I should have changed it.

[chrispsommers]

OK, thanks for clarifying. Thanks for reminding my to create a PyTest!

[mhanif]

Hi @mhanif It seems like the P4Runtime server inside bmv2 is getting a client connection request when another P4Runtime client already has "mastership." The clue is: Not primary. Check to see which containers are running via docker ps in case something is leftover.

Hi @chrispsommers - it seems like that there must be a delay of 10+ seconds when I rerun the same test using "make run-all-tests". I was just doing the "up-arrow" and recalling the command on the Linux prompt. If I wait a little bit and rerun the test, this issue doesn't occur. Seems like the previous incarnation of the P4RT server takes a little bit of time before it cleans up and die. Thanks.

[chrispsommers]

@mhanif you could be right, thanks for the details.

[chrispsommers]

@mukeshmv @mhanif If you get a chance please try out #211, thanks.