Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf: avoid internal object format transformations during export #4385

Open
dyladan opened this issue Dec 20, 2023 · 2 comments
Open

Perf: avoid internal object format transformations during export #4385

dyladan opened this issue Dec 20, 2023 · 2 comments
Labels
Discussion Issue or PR that needs/is extended discussion. never-stale

Comments

@dyladan
Copy link
Member

dyladan commented Dec 20, 2023

Background

Our current internal format for exportable telemetry like spans and metrics (and maybe events) does not match the format expected by the protobuf serialization library we are using. In order to serialize protobuf, this means we have to create new objects with new property names, recursively, for every exported object.

We cannot change the internal representation, because it would be a breaking change for exporter interfaces. Specifically, the SpanProcessor, MetricReader, and Exporter interfaces expect our existing internal formats.

The problem

The overhead for this is quite substantial in a few different ways:

  1. The CPU cost of iterating through every property of every object recursively
  2. The memory cost of creating every object twice (one internal representation and one protobuf representation)
  3. The bundle size cost of including the transformation library, which must contain the full name of every property in both representations
  4. The GC cost of reclaiming the memory for every object in multiple representations

Potential Solutions

  1. Use getters to alias the protobuf names to our internal names. This avoids excess object allocations and cpu iterations at the cost of bundle sizes.
  2. Write a custom protobuf serializer. This would avoid all of the above costs at the expense of developer time and resources. It is likely the best technical choice, but we may not have the resources to take on such a task and guarantee it is updated as OTLP continues to add features.
@dyladan dyladan added the Discussion Issue or PR that needs/is extended discussion. label Dec 20, 2023
@dyladan dyladan added this to the OpenTelemetry SDK 2.0 milestone Dec 20, 2023
Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Discussion Issue or PR that needs/is extended discussion. never-stale
Projects
None yet
Development

No branches or pull requests

2 participants