change label seperator from default comma to options

[andyxning]

Currently, we use hard code comma as the label seperator for concatnating key-value labels. The label string is like:

labels:beta.kubernetes.io/arch:amd64,beta.kubernetes.io/os:linux,kubernetes.io/hostname:127.0.0.1

This is fine until we use Bosun as the alert system and use the group by labels to search for labels.

As currently, bosun use comma to split queried tag key and tag value. For example if the query expression used for query InfluxDB from Bosun is something like this:

$limit = avg(influx("k8s", '''SELECT mean(value) as value FROM "memory/limit" WHERE type = 'node' GROUP BY nodename, labels''', "${INTERVAL}s", "", ""))
$allocatable = avg(influx("k8s", '''SELECT mean(value) as value FROM "memory/node_allocatable" WHERE type = 'node' GROUP BY nodename, labels''', "${INTERVAL}s", "", ""))

Then, we will get tagkey and tag value string like this:

nodename=127.0.0.1,labels=beta.kubernetes.io/arch:amd64,beta.kubernetes.io/os:linux,kubernetes.io/hostname:127.0.0.1

When split by a comma, something wrong happened. we split it wrongly to this:

nodename=127.0.0.1
labels=labels:beta.kubernetes.io/arch:amd64
beta.kubernetes.io/os.linux
kubernetes.io/hostname:127.0.0.1

the last two tag key-value pairs is wrong. This will make bosun confused and panic with something like "panic: opentsdb: bad tag: beta.kubernetes.io/os:linux".

This PR will add a new command line --label-seperator for heapster. And, in order to keep backward compatibility, the default value for --label-seperator is comma. We can specify semicolon as the seperator to walk around the bosun restriction

---

This change is

[k8s-ci-robot]

Can a kubernetes member verify that this patch is reasonable to test? If so, please reply with "@k8s-bot ok to test" on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands will still work. Regular contributors should join the org to skip this step.

If you have questions or suggestions related to this bot's behavior, please file an issue against the kubernetes/test-infra repository.

[andyxning]

@piosz @AlmogBaku @huangyuqi

[piosz]

LGTM, please squash commits before merge

[k8s-ci-robot]

Jenkins GCE e2e failed for commit b088903. Full PR test history.

The magic incantation to run this job again is @k8s-bot test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

[andyxning]

@piosz Done. PTAL.

[AlmogBaku]

Can you add docs? Also I think adding that to metrics will be also useful in this scenario

…

On Thu, 24 Nov 2016 at 0:01 Ning Xie ***@***.***> wrote: @piosz <https://github.com/piosz> done — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1367 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGCpu39WT7iUeDRxy5D2uCzXJlGT3UBks5rBLeigaJpZM4Kqesu> .

[piosz]

+1 to #1367 (comment)

[andyxning]

@AlmogBaku @piosz Doc about --label-seperator command line has been added to docs/storage-schema.md. PTAL.

[piosz]

@k8s-bot ok to test

[piosz]

Thanks @andyxning. I'll merge once I'll be able to make tests green.