Flexibility to increase the EFS TLS Ports via /etc/amazon/efs/efs-utils.conf

[rshiva777]

/kind bug

What happened?
When i run pods with multiple EFS volume mounts, some pods are into pending state due to EFS ports are fully occupied (20049 - 20449)

What you expected to happen?
I expect an way to increase the EFS TLS ports from (20049 - 20449) as per the requirement

How to reproduce it (as minimally and precisely as possible)?

Create a deployment with 10 EFS volume mounts, scale the replica to 50
Anything else we need to know?:

I tried adding a configmap with efs-utils.conf file with updated ports, then mounted it to the csi-node daemonset. Post that i see the below error on daemonset

F0701 15:16:18.687401 1 main.go:65] cannot create config file /etc/amazon/efs for efs-utils. Error: open /etc/amazon/efs/efs-utils.conf: read-only file system

Seems, efs-csi is trying to update the efs-utils.conf file mounted via configmap and getting failed.

Environment

• Kubernetes version (use kubectl version): 1.28
• Driver version: 1.7.3

Please also attach debug logs to help us better diagnose

Defaulted container "efs-plugin" out of: efs-plugin, csi-driver-registrar, liveness-probe
I0701 15:16:18.674262 1 config_dir.go:56] Symlink or directory exists at '/etc/amazon/efs', no need to create one
I0701 15:16:18.674972 1 metadata.go:65] getting MetadataService...
I0701 15:16:18.676846 1 metadata.go:70] retrieving metadata from EC2 metadata service
I0701 15:16:18.685570 1 driver.go:150] Did not find any input tags.
I0701 15:16:18.686302 1 driver.go:116] Registering Node Server
I0701 15:16:18.686420 1 driver.go:118] Registering Controller Server
I0701 15:16:18.686512 1 driver.go:121] Starting efs-utils watchdog
I0701 15:16:18.686885 1 efs_watch_dog.go:221] Skip copying /etc/amazon/efs/efs-utils.conf since it exists already
I0701 15:16:18.686964 1 efs_watch_dog.go:216] Copying /etc/amazon/efs/efs-utils.crt since it doesn't exist
F0701 15:16:18.687401 1 main.go:65] cannot create config file /etc/amazon/efs for efs-utils. Error: open /etc/amazon/efs/efs-utils.conf: read-only file system

• Instructions to gather debug logs can be found here

[daro1337]

I had the same error. In my case, I wanted to change the log level in /etc/amazon/efs/efs-utils.conf, but I am not able to apply my configuration.

[avanish23]

Hi @rshiva777
Where are you getting the 20049 - 20449 port range numbers; officially the ports allowed are 20049 to 21049.
Please share the error you see when the port range is exhausted.

Coming to the questions as to why the port range did not get updated even when you had configured the config maps is because the port range is not configurable and we do not pick the values up from the env variables; as of date only AWS_DEFAULT_REGION and FIPS_ENABLED are configurable and are picked up from the env variables.

@daro1337 even in your case since the log level is currently not configurable you could not set them to DEBUG.
However please refer to the following doc to enable DEBUG logs.

Given this maybe @rshiva777 even you can change the port range by shelling into the driver pod. Please try this and let us know if it works.

Tagging @mskanth972 to confirm the same; can we have a configurable port range and log level via a config map?

[avanish23]

/assign

[mskanth972]

@avanish23, you're right. However, we recently expanded the driver’s port range from 400 to 1000. I suggest trying the latest version, v2.0.8. However, if you're using addons, note that v2.0.8 hasn't been made public yet, so you'll need to use v2.0.7 instead.

The reason you're not seeing any changes after updating the config file is that the EFS CSI Driver uses a separate config file located in watchdog.go. So, even if you modify the EFS Utils config file, it defaults back to the values specified in the watchdog file.

You can refer to this PR for more details: #1402.