Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segfault when enableNetworkPolicy = false and NETWORK_POLICY_ENFORCING_MODE = strict #286

Open
simonlewandowski opened this issue Jul 3, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@simonlewandowski
Copy link

What happened:
We are prepping a migration from calico to vpc-cni for network policy enforcement. We have just encountered a problem where
we set NETWORK_POLICY_ENFORCING_MODE = strict but we are not enabling netpols management yet via vpc-cni policy agent with enableNetworkPolicy = false. This is because we still run calico for netpols in this particular cluster.

Attach logs
segfault from aws-eks-nodeagent

`{"level":"info","ts":"2024-07-03T09:59:36.795Z","caller":"metrics/metrics.go:23","msg":"Serving metrics on ","port":61680}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x55cd067057e2]

goroutine 146 [running]:
github.com/aws/aws-network-policy-agent/pkg/rpc.(*server).EnforceNpToPod(0xc0005e9cc0, {0xc0007b2730?, 0x55cd04fa96c6?}, 0xc0007b2730)
/workspace/pkg/rpc/rpc_handler.go:51 +0x182
github.com/aws/amazon-vpc-cni-k8s/rpc._NPBackend_EnforceNpToPod_Handler({0x55cd074a2680?, 0xc0005e9cc0}, {0x55cd078b2998, 0xc0007d4e40}, 0xc0007a2f80, 0x0)
/go/pkg/mod/github.com/aws/[email protected]/rpc/rpc.pb.go:957 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001c5000, {0x55cd078b2998, 0xc0007d4db0}, {0x55cd078b9fe0, 0xc000876180}, 0xc0007acea0, 0xc000580b10, 0x55cd08ab0ab0, 0x0)
/go/pkg/mod/google.golang.org/[email protected]/server.go:1369 +0xe23
google.golang.org/grpc.(*Server).handleStream(0xc0001c5000, {0x55cd078b9fe0, 0xc000876180}, 0xc0007acea0)
/go/pkg/mod/google.golang.org/[email protected]/server.go:1780 +0x1016
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/go/pkg/mod/google.golang.org/[email protected]/server.go:1019 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 123
/go/pkg/mod/google.golang.org/[email protected]/server.go:1030 +0x135`

What you expected to happen:
aws-eks-nodeagent should start and with given setting it should not enable network policies nor enforce strict mode or segfault.

How to reproduce it (as minimally and precisely as possible):
run vpc-cni with enableNetworkPolicy = false and NETWORK_POLICY_ENFORCING_MODE = strict

Anything else we need to know?:
Another cluster which runs vpc-cni with enableNetworkPolicy = false and NETWORK_POLICY_ENFORCING_MODE = standard did not encounter this crash

Environment:

  • Kubernetes version (use kubectl version): 1.23 (v1.23.17-eks-5e0fdde)
  • CNI Version: amazon-k8s-cni:v1.18.2-eksbuild.1
  • Network Policy Agent Version : aws-network-policy-agent:v1.1.2-eksbuild.1
  • OS (e.g: cat /etc/os-release): Amazon Linux 2
  • Kernel (e.g. uname -a): 5.4.269-183.369.amzn2.x86_64
@simonlewandowski simonlewandowski added the bug Something isn't working label Jul 3, 2024
@adpavlov
Copy link

same for me, but just with

{"env":{"NETWORK_POLICY_ENFORCING_MODE":"strict"}}

@jchen6585
Copy link
Contributor

jchen6585 commented Jul 24, 2024

Hey! Good catch on the issue. The reason for this is that CNI always assumes Network Policy is enabled. When strict mode is set, we hit this code block which sends a request to be processed by the NP agent (even if it is not enabled): https://github.com/aws/amazon-vpc-cni-k8s/blob/master/cmd/routed-eni-cni-plugin/cni.go#L282-L310. We will be tracking a fix for this internally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants