Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky NetworkPolicy NamespaceIsolation tests #5139

Closed
tnqn opened this issue Jun 16, 2023 · 4 comments
Closed

Flaky NetworkPolicy NamespaceIsolation tests #5139

tnqn opened this issue Jun 16, 2023 · 4 comments
Assignees
@Dyanngg
Labels
area/network-policy Issues or PRs related to network policies. kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@tnqn
Copy link
Member

tnqn commented Jun 16, 2023

Describe the bug

The following two runs failed in similar test cases with similar symptons. We should check if it's test's issue or code's issue.

https://github.com/antrea-io/antrea/actions/runs/5286108593/jobs/9565336974
e2e-kind-encap.tar.gz.zip

=== RUN   TestAntreaPolicy/TestGroupNoK8sNP/Case=ACNPNamespaceIsolation
I0616 04:32:43.454815   20150 antreapolicy_test.go:4086] running test case ACNP Namespace isolation for all namespaces
I0616 04:32:43.455180   20150 antreapolicy_test.go:4088] running step Port 80 of test case ACNP Namespace isolation for all namespaces
I0616 04:32:43.455526   20150 k8s_util.go:939] Creating/updating ClusterNetworkPolicy test-acnp-ns-isolation
    antreapolicy_test.go:4661: Waiting for ACNP 'test-acnp-ns-isolation' to be realized
I0616 04:32:43.582371   20150 util.go:45] Confirming ready status costs 110.446821ms
I0616 04:32:49.527814   20150 k8s_util.go:201] y-fayzm7e6/b -> x-fayzm7e6/b: expected Drp but got Con: err - <nil> /// stdout -  /// stderr - TIMEOUT
TIMEOUT

    antreapolicy_test.go:4099: Failure -- 1 wrong results
reachability: correct:80, incorrect:1, result=false

expected:

-	x-fayzm7e6/a	y-fayzm7e6/a	z-fayzm7e6/a	x-fayzm7e6/b	y-fayzm7e6/b	z-fayzm7e6/b	x-fayzm7e6/c	y-fayzm7e6/c	z-fayzm7e6/c
x-fayzm7e6/a	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-fayzm7e6/a	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-fayzm7e6/a	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-fayzm7e6/b	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-fayzm7e6/b	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-fayzm7e6/b	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-fayzm7e6/c	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-fayzm7e6/c	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-fayzm7e6/c	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con


observed:

-	x-fayzm7e6/a	y-fayzm7e6/a	z-fayzm7e6/a	x-fayzm7e6/b	y-fayzm7e6/b	z-fayzm7e6/b	x-fayzm7e6/c	y-fayzm7e6/c	z-fayzm7e6/c
x-fayzm7e6/a	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-fayzm7e6/a	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-fayzm7e6/a	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-fayzm7e6/b	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-fayzm7e6/b	Drp	Con	Drp	Con	Con	Drp	Drp	Con	Drp
z-fayzm7e6/b	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-fayzm7e6/c	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-fayzm7e6/c	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-fayzm7e6/c	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con


comparison:

-	x-fayzm7e6/a	y-fayzm7e6/a	z-fayzm7e6/a	x-fayzm7e6/b	y-fayzm7e6/b	z-fayzm7e6/b	x-fayzm7e6/c	y-fayzm7e6/c	z-fayzm7e6/c
x-fayzm7e6/a	.	.	.	.	.	.	.	.	.
y-fayzm7e6/a	.	.	.	.	.	.	.	.	.
z-fayzm7e6/a	.	.	.	.	.	.	.	.	.
x-fayzm7e6/b	.	.	.	.	.	.	.	.	.
y-fayzm7e6/b	.	.	.	X	.	.	.	.	.
z-fayzm7e6/b	.	.	.	.	.	.	.	.	.
x-fayzm7e6/c	.	.	.	.	.	.	.	.	.
y-fayzm7e6/c	.	.	.	.	.	.	.	.	.
z-fayzm7e6/c	.	.	.	.	.	.	.	.	.


I0616 04:32:50.837223   20150 k8s_util.go:968] Deleting AntreaClusterNetworkPolicies test-acnp-ns-isolation
I0616 04:32:50.852714   20150 antreapolicy_test.go:4086] running test case ACNP Namespace isolation for namespace x
I0616 04:32:50.852763   20150 antreapolicy_test.go:4088] running step Port 80 of test case ACNP Namespace isolation for namespace x
I0616 04:32:50.852776   20150 k8s_util.go:939] Creating/updating ClusterNetworkPolicy test-acnp-ns-isolation-applied-to-per-rule
    antreapolicy_test.go:4661: Waiting for ACNP 'test-acnp-ns-isolation-applied-to-per-rule' to be realized
I0616 04:32:51.017134   20150 util.go:45] Confirming ready status costs 103.085467ms
I0616 04:32:58.5[925](https://github.com/antrea-io/antrea/actions/runs/5286108593/jobs/9565336974#step:8:926)34   20150 k8s_util.go:968] Deleting AntreaClusterNetworkPolicies test-acnp-ns-isolation-applied-to-per-rule

https://github.com/antrea-io/antrea/actions/runs/5225813487/jobs/9435897956?pr=5038
e2e-kind-noencap.tar.gz.zip

=== RUN   TestAntreaPolicy/TestGroupNoK8sNP/Case=ACNPStrictNamespaceIsolation
I0609 20:53:42.715117   20113 antreapolicy_test.go:4009] running test case ACNP strict Namespace isolation for all namespaces
I0609 20:53:42.715145   20113 antreapolicy_test.go:4011] running step Namespace isolation, Port 80 of test case ACNP strict Namespace isolation for all namespaces
I0609 20:53:42.715159   20113 k8s_util.go:939] Creating/updating ClusterNetworkPolicy test-acnp-strict-ns-isolation
    antreapolicy_test.go:4584: Waiting for ACNP 'test-acnp-strict-ns-isolation' to be realized
I0609 20:53:42.844561   20113 util.go:44] Confirming ready status costs 103.872302ms
I0609 20:53:48.789095   20113 k8s_util.go:201] x-itc2ll3l/b -> y-itc2ll3l/b: expected Drp but got Con: err - <nil> /// stdout -  /// stderr - TIMEOUT
TIMEOUT

    antreapolicy_test.go:4022: Failure -- 1 wrong results
reachability: correct:80, incorrect:1, result=false

expected:

-	x-itc2ll3l/a	y-itc2ll3l/a	z-itc2ll3l/a	x-itc2ll3l/b	y-itc2ll3l/b	z-itc2ll3l/b	x-itc2ll3l/c	y-itc2ll3l/c	z-itc2ll3l/c
x-itc2ll3l/a	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-itc2ll3l/a	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-itc2ll3l/a	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-itc2ll3l/b	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-itc2ll3l/b	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-itc2ll3l/b	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-itc2ll3l/c	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-itc2ll3l/c	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-itc2ll3l/c	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con


observed:

-	x-itc2ll3l/a	y-itc2ll3l/a	z-itc2ll3l/a	x-itc2ll3l/b	y-itc2ll3l/b	z-itc2ll3l/b	x-itc2ll3l/c	y-itc2ll3l/c	z-itc2ll3l/c
x-itc2ll3l/a	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-itc2ll3l/a	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-itc2ll3l/a	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-itc2ll3l/b	Con	Drp	Drp	Con	Con	Drp	Con	Drp	Drp
y-itc2ll3l/b	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-itc2ll3l/b	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con
x-itc2ll3l/c	Con	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp
y-itc2ll3l/c	Drp	Con	Drp	Drp	Con	Drp	Drp	Con	Drp
z-itc2ll3l/c	Drp	Drp	Con	Drp	Drp	Con	Drp	Drp	Con


comparison:

-	x-itc2ll3l/a	y-itc2ll3l/a	z-itc2ll3l/a	x-itc2ll3l/b	y-itc2ll3l/b	z-itc2ll3l/b	x-itc2ll3l/c	y-itc2ll3l/c	z-itc2ll3l/c
x-itc2ll3l/a	.	.	.	.	.	.	.	.	.
y-itc2ll3l/a	.	.	.	.	.	.	.	.	.
z-itc2ll3l/a	.	.	.	.	.	.	.	.	.
x-itc2ll3l/b	.	.	.	.	X	.	.	.	.
y-itc2ll3l/b	.	.	.	.	.	.	.	.	.
z-itc2ll3l/b	.	.	.	.	.	.	.	.	.
x-itc2ll3l/c	.	.	.	.	.	.	.	.	.
y-itc2ll3l/c	.	.	.	.	.	.	.	.	.
z-itc2ll3l/c	.	.	.	.	.	.	.	.	.


I0609 20:53:49.1[813](https://github.com/antrea-io/antrea/actions/runs/5225813487/jobs/9435897956?pr=5038#step:8:814)48   20113 antreapolicy_test.go:4011] running step Namespace isolation with K8s NP, Port 80 of test case ACNP strict Namespace isolation for all namespaces
I0609 20:53:49.181383   20113 k8s_util.go:682] Creating/updating NetworkPolicy 'x-itc2ll3l/default-deny-in-namespace-x'
I0609 20:53:49.287001   20113 util.go:44] Confirming ready status costs 100.204193ms
I0609 20:53:55.760406   20113 k8s_util.go:968] Deleting AntreaClusterNetworkPolicies test-acnp-strict-ns-isolation
I0609 20:53:55.764142   20113 k8s_util.go:703] Deleting NetworkPolicy 'x-itc2ll3l/default-deny-in-namespace-x'
@tnqn tnqn added kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. area/network-policy Issues or PRs related to network policies. labels Jun 16, 2023
@tnqn tnqn added this to the Antrea v1.13 release milestone Jun 16, 2023
@tnqn
Copy link
Member Author

tnqn commented Jun 16, 2023

@GraysonWu @Dyanngg

@tnqn tnqn mentioned this issue Jun 16, 2023
Merged
@tnqn
Copy link
Member Author

tnqn commented Jul 13, 2023

another occurrence:
https://github.com/antrea-io/antrea/actions/runs/5538237980/jobs/10108110318?pr=5240

=== RUN   TestAntreaPolicy/TestGroupNoK8sNP/Case=ACNPNamespaceIsolation
I0713 02:26:26.681188   21385 antreapolicy_test.go:4086] running test case ACNP Namespace isolation for all namespaces
I0713 02:26:26.681593   21385 antreapolicy_test.go:4088] running step Port 80 of test case ACNP Namespace isolation for all namespaces
I0713 02:26:26.6[819](https://github.com/antrea-io/antrea/actions/runs/5538237980/jobs/10108110318?pr=5240#step:8:820)45   21385 k8s_util.go:952] Creating/updating ClusterNetworkPolicy test-acnp-ns-isolation
    antreapolicy_test.go:4661: Waiting for ACNP 'test-acnp-ns-isolation' to be realized
I0713 02:26:26.905724   21385 util.go:45] Confirming ready status costs 203.226455ms
I0713 02:26:34.502507   21385 k8s_util.go:981] Deleting AntreaClusterNetworkPolicies test-acnp-ns-isolation
I0713 02:26:34.507689   21385 antreapolicy_test.go:4086] running test case ACNP Namespace isolation for namespace x
I0713 02:26:34.507727   21385 antreapolicy_test.go:4088] running step Port 80 of test case ACNP Namespace isolation for namespace x
I0713 02:26:34.507742   21385 k8s_util.go:952] Creating/updating ClusterNetworkPolicy test-acnp-ns-isolation-applied-to-per-rule
    antreapolicy_test.go:4661: Waiting for ACNP 'test-acnp-ns-isolation-applied-to-per-rule' to be realized
I0713 02:26:34.677556   21385 util.go:45] Confirming ready status costs 102.84503ms
I0713 02:26:40.411278   21385 k8s_util.go:214] x-tanhv5vb/c -> y-tanhv5vb/b: expected Drp but got Err: err - <nil> /// stdout -  /// stderr - 1: TIMEOUT
2: TIMEOUT
3: 
    antreapolicy_test.go:4099: Failure -- 1 wrong results
reachability: correct:80, incorrect:1, result=false

expected:

-	z-tanhv5vb/a	x-tanhv5vb/a	y-tanhv5vb/a	x-tanhv5vb/b	y-tanhv5vb/b	z-tanhv5vb/b	x-tanhv5vb/c	y-tanhv5vb/c	z-tanhv5vb/c
z-tanhv5vb/a	Con	Con	Con	Con	Con	Con	Con	Con	Con
x-tanhv5vb/a	Drp	Con	Drp	Con	Drp	Drp	Con	Drp	Drp
y-tanhv5vb/a	Con	Con	Con	Con	Con	Con	Con	Con	Con
x-tanhv5vb/b	Drp	Con	Drp	Con	Drp	Drp	Con	Drp	Drp
y-tanhv5vb/b	Con	Con	Con	Con	Con	Con	Con	Con	Con
z-tanhv5vb/b	Con	Con	Con	Con	Con	Con	Con	Con	Con
x-tanhv5vb/c	Drp	Con	Drp	Con	Drp	Drp	Con	Drp	Drp
y-tanhv5vb/c	Con	Con	Con	Con	Con	Con	Con	Con	Con
z-tanhv5vb/c	Con	Con	Con	Con	Con	Con	Con	Con	Con


observed:

-	z-tanhv5vb/a	x-tanhv5vb/a	y-tanhv5vb/a	x-tanhv5vb/b	y-tanhv5vb/b	z-tanhv5vb/b	x-tanhv5vb/c	y-tanhv5vb/c	z-tanhv5vb/c
z-tanhv5vb/a	Con	Con	Con	Con	Con	Con	Con	Con	Con
x-tanhv5vb/a	Drp	Con	Drp	Con	Drp	Drp	Con	Drp	Drp
y-tanhv5vb/a	Con	Con	Con	Con	Con	Con	Con	Con	Con
x-tanhv5vb/b	Drp	Con	Drp	Con	Drp	Drp	Con	Drp	Drp
y-tanhv5vb/b	Con	Con	Con	Con	Con	Con	Con	Con	Con
z-tanhv5vb/b	Con	Con	Con	Con	Con	Con	Con	Con	Con
x-tanhv5vb/c	Drp	Con	Drp	Con	Err	Drp	Con	Drp	Drp
y-tanhv5vb/c	Con	Con	Con	Con	Con	Con	Con	Con	Con
z-tanhv5vb/c	Con	Con	Con	Con	Con	Con	Con	Con	Con


comparison:

-	z-tanhv5vb/a	x-tanhv5vb/a	y-tanhv5vb/a	x-tanhv5vb/b	y-tanhv5vb/b	z-tanhv5vb/b	x-tanhv5vb/c	y-tanhv5vb/c	z-tanhv5vb/c
z-tanhv5vb/a	.	.	.	.	.	.	.	.	.
x-tanhv5vb/a	.	.	.	.	.	.	.	.	.
y-tanhv5vb/a	.	.	.	.	.	.	.	.	.
x-tanhv5vb/b	.	.	.	.	.	.	.	.	.
y-tanhv5vb/b	.	.	.	.	.	.	.	.	.
z-tanhv5vb/b	.	.	.	.	.	.	.	.	.
x-tanhv5vb/c	.	.	.	.	X	.	.	.	.
y-tanhv5vb/c	.	.	.	.	.	.	.	.	.
z-tanhv5vb/c	.	.	.	.	.	.	.	.	.


I0713 02:26:41.[830](https://github.com/antrea-io/antrea/actions/runs/5538237980/jobs/10108110318?pr=5240#step:8:831)553   21385 k8s_util.go:981] Deleting AntreaClusterNetworkPolicies test-acnp-ns-isolation-applied-to-per-rule

@tnqn tnqn removed this from the Antrea v1.13 release milestone Jul 14, 2023
@Dyanngg Dyanngg self-assigned this Aug 16, 2023
@tnqn tnqn mentioned this issue Sep 20, 2023
Merged
@tnqn
Copy link
Member Author

tnqn commented Oct 26, 2023

The issue occured again after appending an ending message in the test output via #5507:
https://github.com/antrea-io/antrea/actions/runs/6649206071/job/18067744694?pr=5528.

=== RUN   TestAntreaPolicy/TestGroupNoK8sNP/Case=ACNPStrictNamespaceIsolation
I1026 04:31:31.838300   22433 antreapolicy_test.go:4076] running test case ACNP strict Namespace isolation for all namespaces
I1026 04:31:31.838627   22433 antreapolicy_test.go:4078] running step Namespace isolation, Port 80 of test case ACNP strict Namespace isolation for all namespaces
I1026 04:31:31.838997   22433 k8s_util.go:896] Creating/updating ClusterNetworkPolicy test-acnp-strict-ns-isolation
    antreapolicy_test.go:4651: Waiting for ACNP 'test-acnp-strict-ns-isolation' to be realized
I1026 04:31:32.094191   22433 util.go:45] Confirming ready status costs 204.588893ms
I1026 04:31:38.386568   22433 k8s_util.go:219] z-ta9hdtar/a -> y-ta9hdtar/c: expected Drp but got Err: err - <nil> /// stdout -  /// stderr - 1: TIMEOUT
2: TIMEOUT
3: 
    antreapolicy_test.go:4089: Failure -- 1 wrong results
reachability: correct:80, incorrect:1, result=false

@github-actions GitHub Actions
Copy link
Contributor

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 25, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Dyanngg Dyanngg

Labels
area/network-policy Issues or PRs related to network policies. kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tnqn @Dyanngg