[kube-prometheus-stack] Add upgrade path from stable/prometheus-operator

[fktkrt]

What this PR does / why we need it:

I had to migrate our stable/prometheus-operator to the new chart while keeping the existing PersistentVolume.
To do this I had to perform the steps provided below. After the procedure I was able to access the old time-series data stored in the PV.

Which issue this PR fixes

• fixes [kube-prometheus-stack] Can't find upgrade procedure from stable/prometheus-operator #84

Special notes for your reviewer:

This guide covers a fresh installation of the new chart, so not persisted configurations (e.g. in values.yaml, custom dashboards) can be lost after the process is finished.

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[gkarthiks]

lgtm, let wait for one more reviewer.

[scottrigby]

Thanks 😊 This is great if a user needs to upgrade the release name or namespace.

But given that users can do a full name override to match their existing release, would you be willing to add a similar scenario for retaining PVs for an in-place upgrade?

Also is it clear to everyone what data is being persisted here, and how exactly to do that? Users can persist the grafana subchart user account and dashboard data as well, right?

[scottrigby]

Alternatively, we could merge this as-is, and allow other members of the community to document how to do so for an in-place upgrade. WDYT?

[gkarthiks]

@scottrigby I like your second option. Merge this as-is and support the @fktkrt for his contribution and others (or may be @fktkrt would be interested as well) might be interested in contributing the other documentation.

[scottrigby]

Before this is merged, let's have someone try these steps to ensure they work properly. I'm not sure when I'll get to that but once that's verified by at least one other person we can merge this.

I created #121 to make sure we don't forget to address scenarios and details I asked about above.

[fktkrt]

Just performed it in another cluster, added a missing step and cleaned up the second patch syntax.

[fktkrt]

@m42u Please see my recent changes to this PR based on your comments.
I've omitted deleting the endpoint, because that should be taken care of when removing the service itself.

... there's a volumeName under prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec that might help, but this needs validation.

@desaintmartin, there's this option that might solves your comment, any help is welcomed to help validate it as I have limited time at the moment.

[flouthoc]

@fktkrt Path requires me to uninstall previous chart release wouldn't this cause any downtimes ?

[fktkrt]

@fktkrt Path requires me to uninstall previous chart release wouldn't this cause any downtimes ?

@flouthoc Yes, from to point the old is removed and until the new is installed (1-2m), you won't have metrics.
If you are seeking a zero-downtime way to upgrade, then helm'sfullnameOverride option might be a good fit as others have pointed out, although I don't have any personal experience with it (in terms of PV/PVCs at least).

Do you maybe want to give it a try? :)

[flouthoc]

@fktkrt Thanks i guess 1-2m wont do any damage. I am performing a drill on dummy cluster to record delta for downtime, I guess after that we are good to go. Any reason why is this guide not merged to master yet ?

[fktkrt]

@fktkrt Thanks i guess 1-2m wont do any damage. I am performing a drill on dummy cluster to record delta for downtime, I guess after that we are good to go. Any reason why is this guide not merged to master yet ?

@scottrigby said that first let's wait for a few more successful migration report then we can merge this.
So @flouthoc, can you report one more for us? :)

[shaikatz]

I just did a migration from JSONNET version of kube-prometheus to the helm chart, using those instructions more or less, and it worked perfectly, I think that should get merged :-)

[scottrigby]

We have some good user validation of the steps, so let's merge this. It just needs conflicts to be resolved.

[scottrigby]

I have resolved the conflict

[scottrigby]

Fixed markdownlint on your branch @fktkrt

[llamahunter]

The prometheus.prometheusSpec.serviceMonitorSelector changes with the new chart from release: prometheus-operator to release: kube-prometheus-stack. If you have a lot of ServiceMonitor objects around, do you then need to go through and manually update all your existing ServiceMonitor objects? Is there any way to make it continue to use the old selector? Looking at the helm chart, it didn't seem possible, as the Release.Name is hard coded into the ServiceMonitor objects created by kube-prometheus-stack.

[fktkrt]

I had no issues regarding this with the following setup:

I am using ServiceMonitors and PodMonitors, and labeling those this way:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: foo-podmonitor
spec:
  selector:
    matchLabels:
      app: foo

This is my prometheusSpec:

prometheus:
  prometheusSpec:
    ruleSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

additionally, for my custom environments I am overriding these with externalLabels like this:

prometheus.prometheusSpec.externalLabels.prometheus=dev-prometheus

My Prometheus instance has default labels:

Labels:       app=prometheus
              controller-revision-hash=prometheus-kube-prometheus-stack-prometheus-677954c66f
              prometheus=kube-prometheus-stack-prometheus
              statefulset.kubernetes.io/pod-name=prometheus-kube-prometheus-stack-prometheus-0

and the new release can continue scraping all my existing Pod/ServiceMonitors created with the legacy chart.

If you have access to a test cluster (or can deploy a playground with k3s or similar), you can easily doublecheck it, just to make sure nothing breaks.

[llamahunter]

By turning off the rule/serviceMonitor/podMonitor selectors, doesn't that make it difficult to tell an instance of prometheus to only monitor things that are related to that instance? What if you have multiple prometheuses (promethei?) running in your cluster?

[fktkrt]

You can use additional labels to tie apps to specific prometheus instances.
Another idea that just came up is that if you're using additional charts to manage the CRDs, or you can solve this by modifying only your template of your CRD generator chart.

(By the way, you were close with promethei, https://prometheus.io/docs/introduction/faq/#what-is-the-plural-of-prometheus :) )