Update configuration on add-headers and proxy-set-headers ConfigMap change

[hobti01]

When the ConfigMap specified in add-headers or proxy-set-headers is modified, the nginx configuration should be updated.

Current behavior is to not update nginx configuration when the referenced ConfigMap is modified. This leads to unexpected results when headers are added/modified in the ConfigMap and no change is realized in the running instances. Restarting instances results in a reload of the configuration, but this is not desired.

Previously raised in #3868

/kind feature

[grifx]

Thanks for raising this issue!

What's the appropriate way to handle those kind of issues in a production environment?
Is there a way to manually trigger a configuration build and reload on all the nginx-ingress-controller without restarting the pods?

Thank you!

[longwuyuan]

Does it happen even if you use helm upgrade with a --set for

$ helm inspect all stable/nginx-ingress | grep -i proxySetHeader     
  proxySetHeaders: {}
`controller.proxySetHeaders` | configMap key:value pairs containing [custom headers](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-set-headers) added before sending request to the backends| `{}`
 

[hobti01]

It does not matter how you change the configmap. The functionality does not exist to re-read the configmap. This is a known situation as described in the previous issue where @aledbf requested a new issue (this one). The pods can be restarted, but this feature is to remove that limitation.

Looking forward to 0.31.0!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kmindi]

/remove-lifecycle stale

[ppanyukov]

Guys, what is the workaround for this? I'm hitting this at the moment, and the only way to effect the change is to restart pods. Which is kinda not cool for production, yes?

[ppanyukov]

OK for those using terraform, I found a workaround. The idea is run this command if the values of add-headers changed:

kubectl --namespace ingress rollout restart deployment nginx-ingress-controller

Now in Terraform, the hack is like this:

First, declare headers a s var.

locals {
  ingress_add_headers_yaml = jsonencode({
    "X-Cluster" = "${local.cluster_dns_fqdn}"
    "X-Test" = "this is test 4"
  })
}

Then do the regular helm thing:

resource "helm_release" "ingress" {
  name       = "nginx-ingress"
  namespace  = kubernetes_namespace.ingress.metadata[0].name
  repository = "https://kubernetes-charts.storage.googleapis.com"
  chart      = "nginx-ingress"
  version    = "1.41.2"

  # Use addHeaders defined in a var
  values = [
    <<-EOF
    controller:
      addHeaders: ${local.ingress_add_headers_yaml}
    EOF
  ]
}

And finally, do a shell exec to reload things. Note that this will only run if there are changes in the header.

resource "null_resource" "ingress-reload-changes-in-add-headers" {
  depends_on = [
    helm_release.ingress,
  ]

  triggers = {
    run_if_changed       = sha256(local.ingress_add_headers_yaml)
    cluster_context_name = local.cluster_context_name
    comment              = "it is safe to replace this every time in terraform"
    ingress_add_headers_yaml = local.ingress_add_headers_yaml
    ingress_namespace = kubernetes_namespace.ingress.metadata[0].name
  }

  provisioner "local-exec" {
    # kubectl --namespace ingress rollout restart deployment nginx-ingress-controller
    command = "kubectl --context '${self.triggers.cluster_context_name}' --namespace '${self.triggers.ingress_namespace}' rollout restart deployment nginx-ingress-controller"
  }
}

Not ideal but kinda works.

And it would be nice to have config reload as part of the core functionality so people don't have to hack around it.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kmindi]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[robertvanhoesel]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[robertvanhoesel]

/remove-lifecycle stale

[rohitrav33ndran]

I am having this problem now. I add headers to configmap and the pod restart doesn't update the changes to nginx.conf. Another observation is when multiple add_header statement to configmap and only one is retained on saving the changes.

[strongjz]

@rohitagarwal003 and @hobti01 what version of k8s and inigress-nginx are you using? Does this issue still exist in v0.48.1?

Thanks

/triage needs-information
/priority backlog

[hobti01]

Hi @strongjz , please see the discussion in the PR that was to implement code for this and was not implemented #4030

@aledbf requested that I open a new issue (this one) to capture the result of the previous issue (enhancement request) that has a PR that was not merged.

I'm not aware that the logic has ever been implemented, therefore it would never have been working. Therefore... the versions are irrelevant unless you can point to an implementation :)

[rikatz]

Hi,

As stated in #4030 the implementation should be more simple than that, but we have no intent right now of prioritizing this due to:

• Priority to release ingress-nginx v1 dropping API v1beta1 support
• Fixing long standing time bugs that impact in Ingress performance

I'm open to reviewing some simpler implementation, but right now I don't think we can deal with this, so the issue will be a backlog for medium to long term.

[iamNoah1]

/help

[k8s-ci-robot]

@iamNoah1:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[iamNoah1]

/triage-accepted

[iamNoah1]

/triage accepted

[ppanyukov]

Bump bump. Almost celebrating a 2 year anniversary -- time flies in lockdown! 😀 Would be nice to have a "fix" at some point. 😀

[alokhom]

maybe you want to delete the one pod and check the changes

[iamNoah1]

@alokhom @ppanyukov we are super happy for any contributions :)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[hobti01]

I notice that this is labeled triage/needs-information but it seems that the enhancement is sufficiently described and understood. I think there is a general expectation that all config sources will be re-read when changed. Unfortunately when additional sources (configmaps) are used, this is not the result. While it's great to see the limitation documented, the feature would be even more appreciated. Unfortunately our team hasn't had the capacity to contribute a proposed code change.

/remove-lifecycle stale

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.