Add semantic conventions for cloud provider-specific resource attributes #1099

willarmiros · 2020-10-15T20:00:58Z

Changes

This is an implementation of my proposal in #1098 to include a series of resource attributes that are specific to AWS, and not useful as generalized attributes. I believe this proposal is extendable to include attributes from other cloud providers like GCP, Azure, and any other providers involved in OpenTelemetry. I'd appreciate feedback on anything from the location of this within the spec to the naming of attributes.

One thing syntactically I wasn't sure of was how to describe attributes with common prefixes within a group. For example, for aws.ecs.task.arn and aws.ecs.task.family, should I create a new group called aws.ecs.task with 2 attributes (arn and family), or should I simply include task.arn and task.family as attributes within the larger aws.ecs group, as I've done below?

I've also included a new attribute on the cloud group called infrastructure.service. Please see the modified cloud.md for more details. I think that such an attribute would be useful for users who want to quickly identify the type of compute platform a trace/metric is originating from. I've used the syntax that we use in AWS X-Ray, but I am open to modifying it (e.g. removing the cloud provider prefix since that is already known from cloud.provider attribute, or changing the delimiter). Or I can move the attribute to the AWS spec if others feel it doesn't belong in cloud.md.

Related issue: #968

linux-foundation-easycla · 2020-10-15T20:01:01Z

The committers are authorized under a signed CLA.

✅ William Armiros (b158045)

willarmiros · 2020-10-15T20:01:22Z

/cc @anuraaga @wangzlei

anuraaga · 2020-10-16T05:18:19Z

semantic_conventions/resource/cloud.yaml

@@ -13,7 +13,7 @@ groups:
              brief: 'Amazon Web Services'
            - id: Azure
              value: 'azure'
-              brief: 'Amazon Web Services'
+              brief: 'Microsoft Azure'


anuraaga · 2020-10-16T05:18:32Z

semantic_conventions/resource/cloud.yaml

@@ -37,3 +37,12 @@ groups:
        note: >
          In AWS, this is called availability-zone.
        examples: ['us-central1-a']
+      - id: infrastructure.service


It's probably good to separate this into a separate PR since it's broader reaching than AWS-specific conventions.

Ok will separate it into a new PR with your feedback included.

anuraaga · 2020-10-16T05:20:36Z

semantic_conventions/resource/cloud.yaml

+          The first entry should generally be a cloud provider, followed by a product
+          category, then finally a particular piece of compute infrastructure. Each entry
+          is delimited by a double colon (::).
+        examples: ['AWS::EC2::Instance', 'Azure::Compute::VM', 'GCP::ComputeEngine::VM']


AWS, etc are already in cloud.provider so this should only need something like ec2, aks, cloud run. Of course we'll want insight from a GCP and Azure member for some examples :)

X-Ray uses these extended names but it's the X-Ray exporter that can format the string correctly but not what we'd require in the semantic convention which should just be a precise semantic string.

Makes sense, I sorta figured as much. I'll get rid of the delimiters and just specify the compute platform.

thisthat

Thank you for your contribution :)
I would suggest adding a README.md file inside the aws folder that contains the list of AWS related semantic conventions, similar to what we have for trace semantic conventions :)

thisthat · 2020-10-16T11:41:05Z

semantic_conventions/resource/cloud_provider/aws/aws.yaml

@@ -0,0 +1,34 @@
+groups:


This seems more related to logging than AWS in the general sense.

Would you prefer I move this to somewhere in the logs spec? The loggroup and logstream fields are specific terms to AWS CloudWatch, but if there are more generic terms that would apply to other logging backends but still convey the same meaning, I'd be happy to switch the wording.

I am not familiar with logs, but I guess there will be semantic conventions for it as well!
For the moment, I would suggest changing the id to aws.log to avoid a naming clash with other more general aws semantic conventions.

thisthat · 2020-10-16T11:46:49Z

specification/resource/semantic_conventions/README.md

+
+Attributes that are only applicable to resources from a specific cloud provider. Currently, these
+resources can only be defined for providers listed as a valid `cloud.provider` in
+[Cloud](./cloud.md). Provider-specific attributes all reside in the `cloud_provider` directory.


Maybe add the list of providers for which we have some semantic convention here

Sure, I'll include a list of valid cloud providers more explicitly. I'll also include the README in the aws directory :)

willarmiros · 2020-10-19T21:36:46Z

Made an additional change to make the AWS log attributes to be string arrays. As I noted in the spec:

Multiple log groups must be supported for cases like multi-container applications, where a single application has sidecar containers, and each write to their own log group.

willarmiros · 2020-10-21T16:53:17Z

@thisthat I've applied your suggestions, please give a second look when available :)

thisthat

Thank you for your contribution :)

specification/resource/semantic_conventions/README.md

anuraaga · 2020-10-22T07:45:13Z

Multiple log groups must be supported for cases like multi-container applications, where a single application has sidecar containers, and each write to their own log group.

@willarmiros is this true for telemetry though? Won't any single app process (Resource) only have a single log group since each sidecar container is a different process? I might be confused on it but I think it's appropriate for these to be singular, any metric resource can only be reporting to a single log group I think.

Co-authored-by: Giovanni Liva <[email protected]>

willarmiros · 2020-10-22T16:41:58Z

@anuraaga you're right that each container or process has its own log group, and I suppose I worded it as a feature even though it's more of a limitation. For ECS, the problem is that when calling the Task Metadata Endpoint from the collector container, we have no way of identifying which container is the application container. We just get a list of all containers defined for a task, and we get the container the collector is running on. So we can confidently exclude the collector's log group from the list of log groups, but if there are other sidecar containers we can't distinguish them from the main app container.

Similarly, in EC2, we get the log group data from the agent config file, which can have any number of log groups. Again, we can't distinguish which one is the "right" one. This is why log groups on X-Ray segments are also modeled as an array, even though an X-Ray segment is also only supposed to correspond to one resource.

The only workaround us we get some sort of user input, e.g. in config.yaml. However detector-specific config (e.g. configuration for just the ECS resource detector) doesn't exist and even if we added it, it would still be cumbersome to users IMO. At that point, we might as well just have the user input their log group themselves :)

willarmiros · 2020-10-27T21:32:24Z

@anuraaga when you get a chance a second review would be appreciated :)

anuraaga

Thanks!

…telemetry-specification into aws-conventions

github-actions · 2020-11-06T03:18:35Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

anuraaga · 2020-11-06T05:44:25Z

@open-telemetry/specs-approvers Can anyone take a look at this PR? Thanks!

tigrannajaryan · 2020-11-12T16:46:57Z

@open-telemetry/specs-approvers Can anyone take a look at this PR? Thanks!

@anuraaga I believe we need to resolve #1151 which blocks this.

willarmiros · 2020-11-12T17:35:11Z

@tigrannajaryan I plan on attending the spec issue scrub tomorrow morning to discuss :) if there's a better time/place to raise this please let me know

tigrannajaryan · 2020-11-12T17:40:39Z

@tigrannajaryan I plan on attending the spec issue scrub tomorrow morning to discuss :) if there's a better time/place to raise this please let me know

The spec meeting on Tue may be a better time. Also it would be useful to invite some attention to this issue so that it is discussed offline too. @open-telemetry/specs-approvers please chime in at #1151 if you have thoughts.

willarmiros · 2020-11-12T17:54:05Z

Sounds good. Also to be clear, #1151 blocks #1112 but I don't think it should block this PR. This PR is strictly adding AWS-specific resource fields and adding a place for other cloud-provider-specific attributes to live.

#1112 involves adding a new generic cloud attribute, so understandably in need of more discussion. Regardless I'll plan on attending Tuesday to chat!

hossain-rayhan · 2020-11-17T20:14:07Z

specification/resource/semantic_conventions/cloud_provider/aws/ecs.md

+| Attribute  | Type | Description  | Example  | Required |
+|---|---|---|---|---|
+| `aws.ecs.container.arn` | string | The Amazon Resource Name (ARN) of an [ECS container instance](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_instances.html). | `arn:aws:ecs:us-west-1:123456789123:container/32624152-9086-4f0e-acae-1a75b14fe4d9` | No |
+| `aws.ecs.cluster.arn` | string | The ARN of an [ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html). | `arn:aws:ecs:us-west-2:123456789123:cluster/my-cluster` | No |


When we get task metadata from ECS Task Metadata Endpoint, it gives us the ClusterName instead of ClusterARN for EC2 Launch Type. Can't we add one for aws.ecs.cluster.name?

I originally had it as cluster.name but, since I was populating the ARN for nearly all other fields and consumers could easily derive cluster name from the ARN, I figured recording the cluster ARN made more sense. On Fargate we are provided the cluster ARN by default, and on EC2 launch type, we have the region and account ID from the task ARN, so you can always construct the cluster ARN like I do here.

All this being said, I can still add a new field for the cluster name if you need it. I just wanted to reduce the possibility for redundant info.

I guess very few customers care about the whole ARN. Most useful info here is the resource name (ClusterName or TaskID). It's also implies that all the exporters should write their logic to extract resource name from ARN. It's not only extracting the resource name, they need to do other checks before that. If resource attribute map does not have the name fields, extract them and apply. The logic is not super complex but extra work and check.
So, in a sense, we are blocking our customers if any specific exporter doesn't come with these logics.

Also, I am kind of convinced with the idea of less redundant info. I am OK to follow the experts opinion.

As discussed offline: We're ok with this as is for now, and in the future if customers ask for it we can add cluster name, task ID, etc to this spec.

reyang

LGTM.

proposal for cloud provider-specific resource semantic conventions

b158045

willarmiros requested review from a team October 15, 2020 20:00

github-actions bot assigned tigrannajaryan Oct 15, 2020

anuraaga reviewed Oct 16, 2020

View reviewed changes

thisthat reviewed Oct 16, 2020

View reviewed changes

addressed feedback

9b6c00d

willarmiros mentioned this pull request Oct 19, 2020

added cloud.infrastructure_service attribute to resource spec #1112

Merged

changed log attribs to be string arrays

b8d4c0d

pluralized array attribs

931024a

thisthat approved these changes Oct 22, 2020

View reviewed changes

specification/resource/semantic_conventions/README.md Outdated Show resolved Hide resolved

Update specification/resource/semantic_conventions/README.md

ffb068e

Co-authored-by: Giovanni Liva <[email protected]>

willarmiros mentioned this pull request Oct 23, 2020

Added ECS Resource detector open-telemetry/opentelemetry-collector-contrib#1360

Merged

aabmass mentioned this pull request Oct 26, 2020

fixed more 0.14b0 breakages and updated tests to catch them GoogleCloudPlatform/opentelemetry-operations-python#83

Merged

anuraaga approved these changes Oct 28, 2020

View reviewed changes

willarmiros mentioned this pull request Oct 28, 2020

Define a consistent way to record infrastructure service in use #1151

Closed

William Armiros added 3 commits October 28, 2020 14:01

switched cluster to cluster.arn

c66f5b1

Merge branch 'aws-conventions' of https://github.com/willarmiros/open…

8525215

…telemetry-specification into aws-conventions

lowercased launch types

4024bb0

willarmiros mentioned this pull request Oct 30, 2020

Improved ECS attribute and origin translation in awsxrayexporter open-telemetry/opentelemetry-collector-contrib#1428

Merged

github-actions bot added the Stale label Nov 6, 2020

github-actions bot removed the Stale label Nov 7, 2020

hossain-rayhan reviewed Nov 17, 2020

View reviewed changes

reyang approved these changes Nov 24, 2020

View reviewed changes

Merge branch 'master' into aws-conventions

bcbd99e

carlosalberto merged commit 2aa62df into open-telemetry:master Nov 25, 2020

willarmiros deleted the aws-conventions branch November 25, 2020 16:14

willarmiros mentioned this pull request Jan 14, 2021

Added new semantic conventions open-telemetry/opentelemetry-collector#2365

Merged

willarmiros mentioned this pull request Mar 24, 2021

REQUEST: New membership for willarmiros open-telemetry/community#686

Closed

6 tasks

damemi mentioned this pull request Apr 12, 2023

Add semantic conventions for GCP resources #3384

Closed

damemi mentioned this pull request May 12, 2023

Add GCP Compute Engine resource attributes for host/instance open-telemetry/semantic-conventions#15

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add semantic conventions for cloud provider-specific resource attributes #1099

Add semantic conventions for cloud provider-specific resource attributes #1099

willarmiros commented Oct 15, 2020

linux-foundation-easycla bot commented Oct 15, 2020 •

edited

Loading

willarmiros commented Oct 15, 2020

anuraaga Oct 16, 2020

anuraaga Oct 16, 2020

willarmiros Oct 16, 2020

anuraaga Oct 16, 2020

willarmiros Oct 16, 2020

thisthat left a comment

thisthat Oct 16, 2020

willarmiros Oct 16, 2020

thisthat Oct 19, 2020

thisthat Oct 16, 2020

willarmiros Oct 16, 2020 •

edited

Loading

willarmiros commented Oct 19, 2020

willarmiros commented Oct 21, 2020

thisthat left a comment

anuraaga commented Oct 22, 2020

willarmiros commented Oct 22, 2020

willarmiros commented Oct 27, 2020

anuraaga left a comment

github-actions bot commented Nov 6, 2020

anuraaga commented Nov 6, 2020

tigrannajaryan commented Nov 12, 2020

willarmiros commented Nov 12, 2020 •

edited

Loading

tigrannajaryan commented Nov 12, 2020

willarmiros commented Nov 12, 2020

hossain-rayhan Nov 17, 2020 •

edited

Loading

willarmiros Nov 18, 2020 •

edited

Loading

hossain-rayhan Nov 18, 2020

willarmiros Nov 18, 2020

reyang left a comment

Add semantic conventions for cloud provider-specific resource attributes #1099

Add semantic conventions for cloud provider-specific resource attributes #1099

Conversation

willarmiros commented Oct 15, 2020

Changes

linux-foundation-easycla bot commented Oct 15, 2020 • edited Loading

willarmiros commented Oct 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thisthat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willarmiros Oct 16, 2020 • edited Loading

Choose a reason for hiding this comment

willarmiros commented Oct 19, 2020

willarmiros commented Oct 21, 2020

thisthat left a comment

Choose a reason for hiding this comment

anuraaga commented Oct 22, 2020

willarmiros commented Oct 22, 2020

willarmiros commented Oct 27, 2020

anuraaga left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 6, 2020

anuraaga commented Nov 6, 2020

tigrannajaryan commented Nov 12, 2020

willarmiros commented Nov 12, 2020 • edited Loading

tigrannajaryan commented Nov 12, 2020

willarmiros commented Nov 12, 2020

hossain-rayhan Nov 17, 2020 • edited Loading

Choose a reason for hiding this comment

willarmiros Nov 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyang left a comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Oct 15, 2020 •

edited

Loading

willarmiros Oct 16, 2020 •

edited

Loading

willarmiros commented Nov 12, 2020 •

edited

Loading

hossain-rayhan Nov 17, 2020 •

edited

Loading

willarmiros Nov 18, 2020 •

edited

Loading