flake: `allow-all/pod-to-world/http(s)-to-cilium-io` fails #367

gandro · 2021-06-24T14:53:00Z

Observed in #366 on EKS:

https://github.com/cilium/cilium-cli/pull/366/checks?check_run_id=2905226287

The flow logs indicate that a DNS request was made, but no TCP connection was ever established:

   [.] Action [allow-all/pod-to-world/http-to-www-google: cilium-test/client-7b7bf54b85-h6qvt (10.0.1.205) -> www-google-http (www.google.com:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://www.google.com:80" failed: command terminated with exit code 28
  📄 Matching flows for pod cilium-test/client-7b7bf54b85-h6qvt
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ⌛ Waiting (5s) for flows: Required flows not found yet
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ℹ️  SYN and(ip(src=10.0.1.205),tcp(dstPort=80),tcpflags(syn)) not found
  ✅ DNS request found at 0
  ✅ DNS response found at 3
  ❌ Flow validation failed for pod cilium-test/client-7b7bf54b85-h6qvt: 1 failures (first: 0, last: 3, matched: 2)

It's not clear why there was never any TCP outgoing connection from curl. Unfortunately we don't seem to collect any L7 flows which would give us more insight into what the DNS response was.

The text was updated successfully, but these errors were encountered:

Bring back the DNS visibility annotation (466e8f1) that got reverted in PR #356. I reverted it because it depended on 07a161d which was causing issue #355. I updated the commit so that it no longer depends on 07a161d. Having DNS visibility might give us some additional info to debug #367. Signed-off-by: Jarno Rajahalme <[email protected]> Signed-off-by: Michi Mutsuzaki <[email protected]>

Bring back the DNS visibility annotation (466e8f1) that got reverted in PR #356. I reverted the commit because it depended on 07a161d which was causing issue #355. I modified the commit so that it no longer depends on 07a161d. Having DNS visibility might give us some additional info to debug #367. Signed-off-by: Jarno Rajahalme <[email protected]> Signed-off-by: Michi Mutsuzaki <[email protected]>

tklauser · 2021-06-25T09:50:16Z

Seen the same on GKE with #364: https://github.com/cilium/cilium-cli/pull/364/checks?check_run_id=2903745040

Use [jenkins.]cilium.io for FQDN tests to reduce external dependencies. Ref: #367 Signed-off-by: Michi Mutsuzaki <[email protected]>

tklauser · 2021-07-02T07:54:34Z

#373 changed the connectivity tests to use *.cilium.io instead of *.google.com, but it seems like failures of this kind still occur, see e.g. https://github.com/cilium/cilium-cli/runs/2969863237.

michi-covalent · 2021-07-02T16:45:05Z

#373 changed the connectivity tests to use *.cilium.io instead of *.google.com, but it seems like failures of this kind still occur, see e.g. https://github.com/cilium/cilium-cli/runs/2969863237.

this one is actually even worse, it looks like curl succeeded (no 28 error) but still no SYN packet 🤯

michi-covalent · 2021-08-05T22:07:57Z

still happening with cilium.io: https://github.com/cilium/cilium-cli/actions/runs/1102848558

To avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 Signed-off-by: Martynas Pumputis <[email protected]>

This commit adds an option IPsec e2e upgrade test. The test does the following: * Install the latest stable release version of Cilium. * Run the CLI's connectivity tests. * Upgrade Cilium to a PR version. * Run the CLI's connectivity tests. * Downgrade to the previous version. * Run the CLI's connectivity tests. After each connectivity test case we flush CT to avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 The test runs pods which established long-lived connections. The test checks whether they are not interrupted during upgrade / downgrade. Signed-off-by: Martynas Pumputis <[email protected]>

[ upstream commit e695db5 ] This commit adds an option IPsec e2e upgrade test. The test does the following: * Install the latest stable release version of Cilium. * Run the CLI's connectivity tests. * Upgrade Cilium to a PR version. * Run the CLI's connectivity tests. * Downgrade to the previous version. * Run the CLI's connectivity tests. After each connectivity test case we flush CT to avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 The test runs pods which established long-lived connections. The test checks whether they are not interrupted during upgrade / downgrade. Signed-off-by: Martynas Pumputis <[email protected]>

[ upstream commit e695db5 ] This commit adds an option IPsec e2e upgrade test. The test does the following: * Install the latest stable release version of Cilium. * Run the CLI's connectivity tests. * Upgrade Cilium to a PR version. * Run the CLI's connectivity tests. * Downgrade to the previous version. * Run the CLI's connectivity tests. After each connectivity test case we flush CT to avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 The test runs pods which established long-lived connections. The test checks whether they are not interrupted during upgrade / downgrade. Backport NB: disabled the endpoint routes feature in the 3rd configuration, as the documented limitation was fixed only in >=v1.13. Signed-off-by: Martynas Pumputis <[email protected]>

[ upstream commit e695db5 ] This commit adds an option IPsec e2e upgrade test. The test does the following: * Install the latest stable release version of Cilium. * Run the CLI's connectivity tests. * Upgrade Cilium to a PR version. * Run the CLI's connectivity tests. * Downgrade to the previous version. * Run the CLI's connectivity tests. After each connectivity test case we flush CT to avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 The test runs pods which established long-lived connections. The test checks whether they are not interrupted during upgrade / downgrade. Signed-off-by: Martynas Pumputis <[email protected]>

[ upstream commit e695db5 ] This commit adds an option IPsec e2e upgrade test. The test does the following: * Install the latest stable release version of Cilium. * Run the CLI's connectivity tests. * Upgrade Cilium to a PR version. * Run the CLI's connectivity tests. * Downgrade to the previous version. * Run the CLI's connectivity tests. After each connectivity test case we flush CT to avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 The test runs pods which established long-lived connections. The test checks whether they are not interrupted during upgrade / downgrade. Backport NB: disabled the endpoint routes feature in the 3rd configuration, as the documented limitation was fixed only in >=v1.13. Signed-off-by: Martynas Pumputis <[email protected]>

gandro added the area/CI Continuous Integration testing issue or flake label Jun 24, 2021

gandro mentioned this issue Jun 24, 2021

workflows: Add --timestamps to kubectl logs #366

Merged

michi-covalent mentioned this issue Jun 25, 2021

test: Add DNS visibility annotations to client deployments #369

Closed

michi-covalent added a commit that referenced this issue Jun 25, 2021

check: Replace [www.]google.com with [jenkins.]cilium.io

377eed5

Use [jenkins.]cilium.io for FQDN tests to reduce external dependencies. Ref: #367 Signed-off-by: Michi Mutsuzaki <[email protected]>

michi-covalent mentioned this issue Jun 25, 2021

check: Replace [www.]google.com with [jenkins.]cilium.io #373

Merged

gandro mentioned this issue Jun 29, 2021

Prepare for release v0.8.3 #381

Merged

michi-covalent added a commit that referenced this issue Jun 29, 2021

check: Replace [www.]google.com with [jenkins.]cilium.io

8df1ae6

Use [jenkins.]cilium.io for FQDN tests to reduce external dependencies. Ref: #367 Signed-off-by: Michi Mutsuzaki <[email protected]>

michi-covalent added a commit that referenced this issue Jun 29, 2021

check: Replace [www.]google.com with [jenkins.]cilium.io

34cb7c8

Use [jenkins.]cilium.io for FQDN tests to reduce external dependencies. Ref: #367 Signed-off-by: Michi Mutsuzaki <[email protected]>

nbusseneau mentioned this issue Jul 1, 2021

workflows: do not wait when installing on Kind #386

Merged

nbusseneau pushed a commit that referenced this issue Jul 1, 2021

check: Replace [www.]google.com with [jenkins.]cilium.io

8e93a95

Use [jenkins.]cilium.io for FQDN tests to reduce external dependencies. Ref: #367 Signed-off-by: Michi Mutsuzaki <[email protected]>

nbusseneau mentioned this issue Jul 1, 2021

workflows: check status of Relay port-forward #390

Merged

twpayne mentioned this issue Jul 6, 2021

contrib: Explicitly set remote for backport branches cilium/cilium#16804

Merged

This was referenced Jul 8, 2021

version: report the cilium images version #121

Merged

Run "Post-test information gathering" step on cancellation #426

Merged

nbusseneau mentioned this issue Jul 19, 2021

workflows: refactor EKS workflows #442

Merged

gandro changed the title ~~flake: allow-all/pod-to-world/http-to-www-google fails~~ flake: allow-all/pod-to-world/http(s)-to-cilium-io fails Aug 10, 2021

gandro mentioned this issue Aug 10, 2021

Update Go to 1.16.7 #476

Merged

errordeveloper mentioned this issue Aug 23, 2021

ConformanceEKS (ci-eks) CI: no-policies/pod-to-world/http-to-jenkins-cilium: Fails during connectivity test after deploying with encryption enabled cilium/cilium#16938

Closed

brb added a commit to cilium/cilium that referenced this issue Jun 20, 2023

ci-e2e: Flush CT after upgrade connectivity tests

fd99759

To avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 Signed-off-by: Martynas Pumputis <[email protected]>

brb added a commit to cilium/cilium that referenced this issue Jun 22, 2023

ci-e2e: Flush CT after upgrade connectivity tests

82220f4

To avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 Signed-off-by: Martynas Pumputis <[email protected]>

brb added a commit to cilium/cilium that referenced this issue Jun 27, 2023

ci-e2e: Flush CT after upgrade connectivity tests

46ffca0

To avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 Signed-off-by: Martynas Pumputis <[email protected]>

brb added a commit to cilium/cilium that referenced this issue Jul 10, 2023

ci-e2e: Flush CT after upgrade connectivity tests

015362b

To avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 Signed-off-by: Martynas Pumputis <[email protected]>

brb added a commit to cilium/cilium that referenced this issue Jul 19, 2023

ci-e2e: Flush CT after upgrade connectivity tests

9ccc800

To avoid the interference [1][2] after running L7 proxy tests. [1]: cilium/cilium-cli#367 [2]: #17459 Signed-off-by: Martynas Pumputis <[email protected]>

brb mentioned this issue Jul 21, 2023

ci-ipsec-upgrade: Add IPsec upgrade tests cilium/cilium#26983

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

flake: `allow-all/pod-to-world/http(s)-to-cilium-io` fails #367

flake: `allow-all/pod-to-world/http(s)-to-cilium-io` fails #367

gandro commented Jun 24, 2021 •

edited

Loading

tklauser commented Jun 25, 2021

tklauser commented Jul 2, 2021

michi-covalent commented Jul 2, 2021

michi-covalent commented Aug 5, 2021

flake: allow-all/pod-to-world/http(s)-to-cilium-io fails #367

flake: allow-all/pod-to-world/http(s)-to-cilium-io fails #367

Comments

gandro commented Jun 24, 2021 • edited Loading

tklauser commented Jun 25, 2021

tklauser commented Jul 2, 2021

michi-covalent commented Jul 2, 2021

michi-covalent commented Aug 5, 2021

flake: `allow-all/pod-to-world/http(s)-to-cilium-io` fails #367

flake: `allow-all/pod-to-world/http(s)-to-cilium-io` fails #367

gandro commented Jun 24, 2021 •

edited

Loading