[bitnami/elasticsearch] Add support for IAM Roles for Service Accounts (IRSA)

[jim-barber-he]

Name and Version

bitnami/elasticsearch:8.6.2

What is the problem this feature will solve?

IAM Roles for Service Accounts (IRSA) is a way to grant IAM permissions at a granular container level instead of needing to grant the permissions to the hosts that the containers run on.
It is a superior solution to tools like kiam and kube2iam.

IRSA's pod-identity-webhook admission controller mutates pods to inject environment variables such as AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE into the containers, and defines a volume in the pod with the token and has the containers mount it.
Standard authentication with the AWS SDK will use these environment variables appropriately, however Elasticsearch has its own approach so setting these environment variables isn't enough.

The $AWS_WEB_IDENTITY_TOKEN_FILE variable is set to /var/run/secrets/eks.amazonaws.com/serviceaccount/token but this will not be read by Elasticsearch.
Instead they look for a symlink in the plugin directory as per their code below:
https://github.com/elastic/elasticsearch/blob/8.6/modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java#L315

This approach is also mentioned in their documentation here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html#iam-kubernetes-service-accounts

People not using the Bitnami images and Helm charts create the symlink via an initContainer something like what is shown in this comment linked below:
elastic/elasticsearch#52625 (comment)
NOTE: The comment talking about needing to set a AWS_ROLE_SESSION_NAME environment variable is no longer true as that is handled by Elasticsearch's code these days:
https://github.com/elastic/elasticsearch/blob/8.6/modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java#L336

As far as I can tell there is no way to do something equivalent with the Bitnami images and helm charts (although I could be wrong).

• The plugin area of the filesystem is not on a volume that I can mount into an initContainer?
  So the same approach shown above to have an initContainer create a symlink will not work as it won't be there on the running container's image.

• You can't mount symlinks into a filesystem via the subPath part of a volumeMount, so I don't think I can do any tricks with extraVolumes and extraVolumeMounts.

• I don't think mounting scripts in /docker-entrypoint-initdb.d to create the symlink isn't going to help either since they only run once and not every time a container starts?

  I'm looking for a way for the bitnami/elasticsearch to be able to support the use of IRSA by allowing me to create the symlink in the plugin area.

What is the feature you are proposing to solve the problem?

I guess a couple of approaches I can think of are:

• Something added to one of the start up scripts in the image that has a check for if the $AWS_WEB_IDENTITY_TOKEN_FILE environment variable is set and if so, create the symlink.
  I don't know which script would be most appropriate for this to be added to; maybe /opt/bitnami/scripts/libelasticsearch.sh and then a call to it in /opt/bitnami/scripts/elasticsearch/setup.sh ?

OR

• Provide something similar to being able to mount in shell scripts (like /docker-entrypoint-initdb.d/* that run every time the container starts, not just the first time).

Also I'm not sure where the symlink would need to be created in the bitnami image.
On the official Elasticsearch image it is created as /usr/share/elasticsearch/config/repository-s3/aws-web-identity-token-file
But for the bitnami image I suspect it'll either be /bitnami/elasticsearch/plugins/repository-s3/aws-web-identity-token-file or /opt/bitnami/elasticsearch/config/repository-s3/aws-web-identity-token-file ?

What alternatives have you considered?

I've been looking at the image and chart and I haven't found an obvious way for me to create the symlink in the container's file system that points the the AWS web identity token file.

[jim-barber-he]

After sleeping on this I've thought of a way of achieving what I need using a mix of initContainers and the extraVolumes and extraVolumeMounts settings for the Helm chart.
And I've determined that Elasticsearch will look for the symlink for the token at /opt/bitnami/elasticsearch/config/repository-s3/aws-web-identity-token-file

Here's what I added to the chart:

extraVolumeMounts:
  # Mount the volume containing the IRSA symlink to where Elasticsearch will look for it.
  - mountPath: /opt/bitnami/elasticsearch/config/repository-s3
    name: repository-s3

extraVolumes:
  - emptyDir: {}
    name: repository-s3

initContainers:
  # initContainer for all pods to create a symlink on the repository-s3 volume pointing
  # to the location stored in the $AWS_WEB_IDENTITY_TOKEN_FILE environment variable.
  # This volume is mounted to the appropriate path via the `extraVolumeMounts` settings.
  - command:
      - /bin/sh
      - -c
      - |
        mkdir -p /repository-s3;
        ln -s $AWS_WEB_IDENTITY_TOKEN_FILE /repository-s3/aws-web-identity-token-file
    image: debian:bullseye-slim
    name: symlink-token
    volumeMounts:
      - name: repository-s3
        mountPath: /repository-s3

After that I was able for go to the Dev Tools section of Kibana and run this:

POST /_snapshot/s3-snapshots
{
  "type" : "s3",
  "settings" : {
    "bucket" : "hetest-elasticsearch",
    "chunk_size" : "1g",
    "server_side_encryption" : "true",
    "max_restore_bytes_per_sec" : "1000mb",
    "storage_class" : "standard_ia",
    "readonly" : "true",
    "buffer_size" : "100mb",
    "max_snapshot_bytes_per_sec" : "600mb"
  }
}

When the IAM roles were not in place this resulted in an error trying to use the AWS meta-data service.
But with the configuration above it runs successfully.
I have more testing to do, but so far so good.

I guess this feature request can be closed, OR something could be done to make the user of IRSA easier to achieve.

[CeliaGMqrz]

Hi @jim-barber-he,

Sorry for the delay. Thanks for the detailed information. It looks like a very interesting new feature, but maybe it is too specific. We're a small team and our capacity is not too high. This is currently in our radar but the priority is not too high. Said that, we can not guarantee any ETA.

Anyway, would you like to contribute by creating a PR to solve the issue? The Bitnami team will be happy to review it and provide feedback. Here you can find the contributing guidelines.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.