Relax format accepted for the "OTEL_RESOURCE_ATTRIBUTES" environment variable

[seh]

Problem Description

At present, using the "OTEL_RESOURCE_ATTRIBUTES" environment variable is more difficult than it should be. The prescribed format for the variables values borrows the W3C Baggage HTTP header format, which is approximately key/value pairs conjoined with equals signs, with interleaved commas separating each pair, accommodating some forms of whitespace liberally.

This is easy to type, easy to parse, and easy to compose in a programming language with a capable join function—one that projects a sequence of strings to single string with an interleaved delimiter. However, I find that I often try to compose values for the "OTEL_RESOURCE_ATTRIBUTES" variable in environments in which I don't have such a join function available, where I'm trying to stitch together key/value pairs composed by different parties.

Consider Kubernetes pod specs, where we define the environment variables supplied to each container in the pod. Kubernetes can interpolate placeholders for environment variables that appear in various places—including in other environment variable values. I can use a mutating admission Webhook to inject some OpenTelemetry Resource attribute values into the "OTEL_RESOURCE_ATTRIBUTES" environment variable, but maybe I'd like to combine those with values from other environment variables available in the same container specification.

Consider the following hypothetical configuration fragment:

containers:
- name: exporter
  env:
  # Some elided ...
  # This one establishes some base values that may be overridden.
  - name: OTEL_RESOURCES_ATTRIBUTES_PREFIX
    value: app.color=blue,k8s.deployment.name=something
  # This one contains attributes injected via the Kubernetes downward API.
  - name: OTEL_RESOURCE_ATTRIBUTES_DOWNWARD_API
    value: >-
      k8s.namespace.name=$(POD_NAMESPACE),
      k8s.node.name=$(NODE_NAME),
      k8s.pod.name=$(POD_NAME),
      k8s.pod.primary_ip_address=$(POD_IP_ADDRESS),
      k8s.pod.service_account.name=$(POD_SERVICE_ACCOUNT_NAME),
      k8s.pod.uid=$(POD_UID),
      net.host.ip=$(NODE_IP_ADDRESS)
  # This one overrides anything proposed earlier.
  - name: OTEL_RESOURCE_ATTRIBUTES_SUFFIX
    value: app.style=fancy,app.color=pink
  # Now attempt to glue them all together.
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: >-
      $(OTEL_RESOURCES_ATTRIBUTES_PREFIX)
      $(OTEL_RESOURCE_ATTRIBUTES_DOWNWARD_API)
      $(OTEL_RESOURCE_ATTRIBUTES_SUFFIX)

We have three environment variables we'd like to join into a final "OTEL_RESOURCE_ATTRIBUTES" value. Assume that either or both of the "OTEL_RESOURCES_ATTRIBUTES_PREFIX" and the "OTEL_RESOURCES_ATTRIBUTES_SUFFIX" values may be empty. If we are certain that the intervening "OTEL_RESOURCE_ATTRIBUTES_DOWNWARD_API" value will never be empty, we can mandate that "OTEL_RESOURCES_ATTRIBUTES_PREFIX" is empty or it ends with a comma. Likewise, we can mandate that "OTEL_RESOURCES_ATTRIBUTES_SUFFIX" is empty or it starts with a comma. However, if we intend to concatenate any two values that may both be empty or not, we cannot decide whether we need a trailing comma in the first value, a leading comma in the second, or an intervening comma between them.

What if we could join these fragments together with more confidence and less care? What if we could add leading and trailing commas, and add intervening commas, not worrying about what's in the prefix or suffix fragments?

Proposal

Relax the SDK's enforcement of the W3 Baggage header format, tolerating the following:

• a leading comma
• a trailing comma
• absent "list-members" between commas

At present, the W3 Baggage header format's "baggage-string" production reads as follows:

baggage-string         =  list-member 0*179( OWS "," OWS list-member )

I propose that we accept the following instead:

otel-resource-string   =  *1list-member 0*179( OWS "," *1( OWS list-member ) )

That is, the string can start with a comma, and the next "list-member" after each comma is optional, which also allows the string to end with a comma. Using the split function common in programming language libraries, this requires dropping empty elements from the sequence projected from the input string.

Compatibility Concerns

This proposal expands the universe of valid values for the "OTEL_RESOURCE_ATTRIBUTES" environment variable. All existing values that are valid under the current specification version (1.7.0 as of this writing) remain valid with this proposed change. However, some values that had been invalid before would become newly valid.

Some users may prefer more stringent parsing of these values, such that their application would fail earlier if the attempted composition of this value injected any empty attribute key/value pairs accidentally. I consider the opportunity for and benefit from being able to compose these values more easily—especially in environments in which procedural manipulation isn't feasible—to outweigh this cost.

SDK Implementation Survey

At present, SDKs differ in how they validate candidate values read from the "OTEL_RESOURCE_ATTRIBUTES" environment variable.

Language	Tolerates Leading Comma	Tolerates Missing K/V Pair	Tolerates Trailing Comma
Go	🚫	🚫	🚫
Java	✅	✅	✅
TODO: ...

History

I first raised this topic for discussion in the "otel-specification" of the CNCF Slack workspace on 12 September 2021. I also brought it up for discussion in the OpenTelemetry Go SIG weekly meeting on 30 September 2021.