Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

faas.id incorrectly used as span attribute #442

Closed
dyladan opened this issue Apr 22, 2021 · 11 comments
Closed

faas.id incorrectly used as span attribute #442

dyladan opened this issue Apr 22, 2021 · 11 comments
Assignees
Labels
bug Something isn't working never-stale priority:p2 Bugs and spec inconsistencies which cause telemetry to be incomplete or incorrect

Comments

@dyladan
Copy link
Member

dyladan commented Apr 22, 2021

faas.id is used as a span attribute here, but should be a resource attribute. I left it here to make this PR as small as possible while passing and changing no behavior. I will create a follow up bug to fix this.

Originally posted by @dyladan in #440 (comment)

@dyladan
Copy link
Member Author

dyladan commented Apr 22, 2021

/cc @anuraaga @willarmiros

@dyladan dyladan added the bug Something isn't working label Apr 22, 2021
@dyladan
Copy link
Member Author

dyladan commented Apr 22, 2021

@willarmiros mind if I assign this to you?

@willarmiros
Copy link
Contributor

@dyladan Sure!

@willarmiros
Copy link
Contributor

willarmiros commented Apr 23, 2021

It turns out this is actually a known issue. The faas.id cannot be a resource attribute because resource attributes must be set at initialization time and are thereafter immutable. However a Lambda function's ARN (which is its faas.id) is only readable from the context of a function invocation. So it is impossible to assign it at initialization time and is made a span attribute as a workaround instead.

Here's the original spec issue pointing this out: open-telemetry/opentelemetry-specification#1261

This would either be solved by:

  1. Getting the Lambda team to expose the ARN at startup (we're trying!), or
  2. Adding resource attributes post-creation (e.g. via auto-discovery) opentelemetry-specification#1298, which would allow some resource attribs (like faas.id) to be mutable

@github-actions
Copy link
Contributor

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

@dyladan
Copy link
Member Author

dyladan commented Dec 8, 2021

@willarmiros looks like there has been no spec movement on this since trace GA. Think now might be a good time to try to tackle this?

@willarmiros
Copy link
Contributor

We're talking about related items in the sampling SIG with respect to exposing resource attributes to samplers to inform sampling decisions & whether the Resources should be provided for every sampling decision or just at Sampler creation - I'll bring up this issue at the next sampling SIG

@dyladan
Copy link
Member Author

dyladan commented Sep 21, 2022

@willarmiros any update here?

@dyladan dyladan added the priority:p2 Bugs and spec inconsistencies which cause telemetry to be incomplete or incorrect label Sep 21, 2022
@willarmiros
Copy link
Contributor

Unfortunately due to internal limitations Lambda is not able to expose its ARN at startup. So it seems we are blocked on open-telemetry/opentelemetry-specification#1298 to get this fixed.

@Flarna
Copy link
Member

Flarna commented Sep 22, 2022

Meanwhile spec allows to set faas.id as span attribute instead as resource attribute: On some cloud providers, it may not be possible to determine the full ID at startup, so consider setting faas.id as a span attribute instead.

For AWS it would be possible to set it on resource by delaying resource detection until first invocation but on e.g. azure a single OS process (and therefore single global TracerProvider) may host several functions which have different ids therefore resource is not applicable at all.

see: open-telemetry/opentelemetry-specification#2502

@dyladan
Copy link
Member Author

dyladan commented Sep 26, 2022

interesting I missed that wording in the spec. thanks for clarifying. i'll close this then

@dyladan dyladan closed this as completed Sep 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working never-stale priority:p2 Bugs and spec inconsistencies which cause telemetry to be incomplete or incorrect
Projects
None yet
Development

No branches or pull requests

3 participants