Label addition to metrics with prefix "container.googleapis.com" should not break GCP/GKE?

[wenjiaswe]

Currently, label addition will break GKE. I checked https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd, for metrics with prefix "container.googleapis.com" (e.g., etcd metrics), if definition (for example, label is one of the definition) of the metric was changed, then the metric is marked as broken and the metric is not going to be pushed:

k8s-stackdriver/prometheus-to-sd/translator/metric_descriptor_cache.go

Line 72 in a39287c

func (cache *MetricDescriptorCache) ValidateMetricDescriptors(metrics map[string]*dto.MetricFamily, whitelisted []string) {

. prometheus-to-sd only UpdateMetricDescriptors if it contains "custom.googleapis.com" prefix:

k8s-stackdriver/prometheus-to-sd/main.go

Line 141 in 1047589

if strings.HasPrefix(commonConfig.GceConfig.MetricsPrefix, customMetricsPrefix) {

.

How ever, many label additions are conceptually backward compatible since their introduction would not break existing readers that are unaware of the label. So I would expect backward compatible label addition does not break GCP/GKE.

Here is an example: etcd-io/etcd#10022