Track request body size in XHR and Fetch instrumentations #4706

MustafaHaddara · 2024-05-14T16:00:24Z

Which problem is this PR solving?

The Fetch and XHR instrumentations expose http.response_content_length attributes but do not expose http.request_content_length attributes. This PR adds the http.request_content_length attributes to outgoing requests that have a body (ex. POST, PATCH, DELETE, etc.)

Short description of the changes

Ideally, there would be some browser API would could just read for this (similar to how we get the response content length via the PerformanceObserver API). However, no such API exists.

Second best would be if we could read the content-length request header. Unfortunately, the XMLHTTPRequest API does not offer any way to read request headers. Even if we could (ie. with the fetch API), this header seems to be set automatically by the browser before it actually sends the request, outside of user-space.

So, we have to compute the body length on our own. This PR implements that.

Detailed Description

The first few commits (e349fa4...eaf9786) are refactorings/updates, mainly to unit tests, to enable changes and tests that follow.

The primary changes are contained in these 3 commits:

d6149ca adds getXHRBodyLength and getFetchBodyLength utils to the opentelemetry-sdk-trace-web package.
- getFetchBodyLength needs to call getXHRBodyLength, otherwise I would have defined these in their respective packages.
d97b02b calls getXHRBodyLength from the XHR instrumentation package and adds unit tests for the XHR instrumentation
860557e calls getFetchBodyLength from the Fetch instrumentation package and adds unit tests for the Fetch instrumentation
bee76c8 makes this functionality opt-in

The getXHRBodyLength function is mostly straightforward; the XHR API is not too complicated and is fairly self-explanatory.

On the other hand, the getFetchBodyLength function is more complex. The root of the problem is that the fetch API doesn't expose clean ways for us to get the body content. In places where it is possible, it is often consumable only once, and often as aPromise that resolves to the body content. I had to take care to not consume the actual body content; we do not want this instrumentation to interfere with actual requests. It is possible that a bug in this implementation would result in the bodies on fetch requests getting consuming by this instrumentation and then not actually included in the network request.

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Added unit tests to opentelemetry-sdk-trace-web, opentelemetry-instrumentation-xml-http-request, and opentelemetry-instrumentation-fetch

Checklist:

Followed the style guidelines of this project
Unit tests have been added

packages/opentelemetry-sdk-trace-web/src/utils.ts

codecov · 2024-05-22T19:50:06Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.17%. Comparing base (6f4f3fc) to head (8d6705f).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4706   +/-   ##
=======================================
  Coverage   93.17%   93.17%           
=======================================
  Files         315      315           
  Lines        8086     8086           
  Branches     1617     1617           
=======================================
  Hits         7534     7534           
  Misses        552      552

…y-sdk-trace-web

- webworkers don't have DOMParser or Document APIs - hard-coding an expected length for Document length is fragile (and varies across platforms)

MustafaHaddara · 2024-08-12T16:58:41Z

@scheler @MSNev I've made the changes we discussed and resolved merge conflicts. Please let me know if you have any other questions.

MustafaHaddara · 2024-08-23T20:07:04Z

packages/opentelemetry-sdk-trace-web/src/utils.ts

+function getFormDataSize(formData: FormData): number {
+  let size = 0;
+  for (const [key, value] of formData.entries()) {
+    size += key.length;
+    if (value instanceof Blob) {
+      size += value.size;
+    } else {
+      size += value.length;
+    }
+  }
+  return size;
+}


This is an improvement over the old implementation (which was stringifying the FormData, and that wouldn't actually stringify file objects correctly), but it still doesn't calculate the exact FormData size. Browsers will serialize FormData in their own way (ex including a separator for each of the fields, etc.)

I will keep investigating for more accurate approaches.

Yeah, this seems good enough for now, and maybe if anyone is sufficiently motivated they can contribute the changes to calculate it more precisely for their platform 😬

packages/opentelemetry-sdk-trace-web/src/utils.ts

…o request-body-size

MustafaHaddara · 2024-08-26T19:29:18Z

@JamieDanielson

It could change to http.request.body.size in an upcoming SemConv update, but may not since that attribute is not yet stable. TBD I think. I'm good with the attribute as-is in this PR.

I went ahead and used the new (unstable) attribute; let me know if you want me to revert that.

@tbrockman @MSNev @JamieDanielson I've applied all of the requested changes and this PR should be ready for another review. Let me know if I missed anything.

…o request-body-size

tbrockman · 2024-09-06T14:28:11Z

packages/opentelemetry-sdk-trace-web/src/utils.ts

+    size += key.length;
+    if (value instanceof Blob) {
+      size += value.size;
+    } else {
+      size += value.length;
+    }


Sorry for the 👻 here, just been a bit busy with life and quitting #dayjob!

Just noticed, should these also use getByteLength?

Feels silly to suggest it given that FormData size varies (as you mentioned, browser/platform-specific implementation differences from things like boundaries and such, I just checked out Firefox for example), so I understand if it seems unnecessary at this point since it'd just be shaving a tiny bit of hypothetical inaccuracy off an estimate that will inherently be incorrect under the circumstances that it applies to (and feel free to ignore).

Suggested change

size += key.length;

if (value instanceof Blob) {

size += value.size;

} else {

size += value.length;

}

size += getByteLength(key.length);

if (value instanceof Blob) {

size += value.size;

} else {

size += getByteLength(value.length);

}

tbrockman · 2024-09-06T15:43:27Z

packages/opentelemetry-sdk-trace-web/src/utils.ts

+  }
+
+  if (typeof body === 'string') {
+    return body.length;


@MustafaHaddara For the most part I think that would be absolutely true! But I also wouldn't underestimate the potential for people to send large JSON payloads (either intentionally or unintentionally).

As a contemporary example, OpenAI's chat API accepts user image input as "[e]ither a URL of the image or the base64 encoded image data.", so you can imagine that for a large image this might amount to some meaningful overhead if someone sends their images inline.

Given some of the other inherent issues for calculating content length size with FormData, would you be open to allowing users to specify their own optional getXHRBodyLength (or maybe exposed as calculateBodyLength) function?

For context, I'm hoping to use this functionality in Browser Extension for OpenTelemetry, and I'd like to limit the overhead as much as possible when it already involves injecting big blobs of Javascript into pages which might be making requests that extension users may not have much control over. This way, I can supply my own implementation (which may be less maintainable/correct/costly) and also experiment with browser-specific calculations, without any maintenance burden being placed on this project.

JamieDanielson · 2024-09-25T19:24:36Z

from comment thread above, moving here for easier finding:

In the semantic conventions tooling meeting this morning a couple of things came up which affect what this PR should be doing

The http.request_content_length header is actually deprecated in favor of http.request.body.size in v1.21.0

The http.request.body.size is listed as experimental with a requirement level of Opt-In which implies that all usages SHOULD be behind an experimental flag and not added by default.

Follow-up notes from JS SIG meeting discussion:

This PR just uses http.request.body.size since the previous version http.request_content_length didn't exist previously anyway, which should be fine.
This PR does have an opt-in mechanism, and it is disabled by default.
Because this opt-in mechanism for certain http attributes includes more than just this attribute and this instrumentation, it may be preferred to move this configuration flag elsewhere so it can be shared by multiple instrumentations. For example, Java added an experimental-opt-in flag that includes these.

@dyladan @MSNev did I capture that correctly? And if so, what are the next steps here? I suspect we'll keep much of this logic already written in this PR and just change the enable/disable mechanism.

MustafaHaddara · 2024-10-15T13:45:10Z

@dyladan @MSNev I just wanted to follow up on @JamieDanielson's message above-- what would you like me to do with this PR?

MustafaHaddara · 2024-10-16T16:16:54Z

We discussed this PR at the client-side SIG yesterday. The two remaining discussion threads:

from @JamieDanielson:

Because this opt-in mechanism for certain http attributes includes more than just this attribute and this instrumentation, it may be preferred to move this configuration flag elsewhere so it can be shared by multiple instrumentations. For example, Java added an experimental-opt-in flag that includes these.

My position is that we can update the opt-in mechanism in the future, if we decide to go that route and build one flag to control multiple instrumentations.

As for @tbrockman 's memory concerns:

But I also wouldn't underestimate the potential for people to send large JSON payloads (either intentionally or unintentionally).

[...]

Given some of the other inherent issues for calculating content length size with FormData, would you be open to allowing users to specify their own optional getXHRBodyLength (or maybe exposed as calculateBodyLength) function?

I'm not too worried about memory concerns since this entire instrumentation is opt-in, and I think making the suggested change is something that we can discuss in a follow up issue or PR.

pichlermarc · 2024-10-23T11:26:08Z

experimental/packages/opentelemetry-instrumentation-xml-http-request/README.md

+| `http.scheme`                               | The URI scheme identifying the used protocol                                   |
+| `http.url`                                  | Full HTTP request URL                                                          |
+| `http.method`                               | HTTP request method                                                            |
+| `http.request.body.size`                    | Uncompressed size of the request body, if any body exists                      |


This instrumentation implement Semantic Conventions v1.7 so we'd have to emit http.request_content_length instead to avoid inconsistency in the emitted telemetry.

I know we mentioned earlier that this should be fine but we have rejected/requested changes on PRs (for example #4962) in the past for similar kinds of inconsistency.

Requested changes: using http.request_content_length over the latest semantic conventions (I think it was mentioned that the attribute did not exist in the semconv package in an earlier comment, if that's indeed the case we should introduce a local constant in the respective instrumentation package to use it)

hmm fair point. I think this should actually be http.request_content_length_uncompressed though? Since we measure the raw payload size, before it gets gzip'd by the browser.

I've used that attribute name, let me know if I should switch to the compressed attribute instead.

pichlermarc · 2024-10-23T11:45:50Z

packages/opentelemetry-sdk-trace-web/src/utils.ts

@@ -125,6 +129,155 @@ export function addSpanNetworkEvents(
  }
 }

+function _getBodyNonDestructively(body: ReadableStream) {


My guess is that the reason this requires changes to @opentelemetry/sdk-trace-web is that we're trying to:

reduce added bundle size for when both instrumentations are added

reduce duplicated code and tests for the same functionality.

I wonder though: is the added public API surface in a stable package worth the trade-off? I think since it's only duplicated twice the vast majority of the end-users will never use this function directly, I'd much rather have duplicated code than the added public stable API that we have to keep stable in perpetuity.

Q: given that these instrumentations share code so directly, and the concepts they instrument are so intertwined, would it make sense for these instrumentations to be located in a single package instead of two different ones? Both of the packages are experimental so moving them around is not completely impossible. I'm also open to other ideas 🙂

My guess is that the reason this requires changes to @opentelemetry/sdk-trace-web is that we're trying to:

reduce added bundle size for when both instrumentations are added

reduce duplicated code and tests for the same functionality.

Yeah that's exactly correct. I can duplicate the logic across both packages if you'd prefer that.

I think it could make sense to unify the two packages. Both packages deal with network instrumentation and, given the state of the web, I would assume that most projects have a mix of fetch and XHR calls in their code. I would be surprised if there were a lot of people intentionally using only one of the 2 instrumentations.

But unifying them is a higher level decision that I'll defer to you and the other maintainers :)

MustafaHaddara · 2024-11-05T17:10:43Z

@pichlermarc I've responded to your comments and made the changes you requested. Are you able to take a look?

…o request-body-size

MustafaHaddara requested a review from a team May 14, 2024 16:00

MustafaHaddara force-pushed the request-body-size branch 3 times, most recently from a4cb688 to 14c9323 Compare May 15, 2024 14:54

MSNev reviewed May 21, 2024

View reviewed changes

packages/opentelemetry-sdk-trace-web/src/utils.ts Outdated Show resolved Hide resolved

scheler suggested changes May 21, 2024

View reviewed changes

packages/opentelemetry-sdk-trace-web/src/utils.ts Show resolved Hide resolved

packages/opentelemetry-sdk-trace-web/src/utils.ts Outdated Show resolved Hide resolved

MustafaHaddara force-pushed the request-body-size branch 2 times, most recently from 0988e15 to c67e910 Compare May 22, 2024 19:45

MustafaHaddara force-pushed the request-body-size branch 2 times, most recently from 7351d79 to 62d5b82 Compare June 12, 2024 20:11

MustafaHaddara requested review from scheler and MSNev June 12, 2024 20:14

MustafaHaddara force-pushed the request-body-size branch from 62d5b82 to 4540ad0 Compare June 25, 2024 16:04

MustafaHaddara force-pushed the request-body-size branch from 26ea752 to 92da8c9 Compare July 16, 2024 15:52

MustafaHaddara added 12 commits August 12, 2024 12:54

reduce overlap between (variable shadowing) between individual tests

e349fa4

update xhr test to check attr names directly instead of using key order

656cbc4

update fetch test to read attributes directly instead of using key order

538e712

override api request in beforeEach call

eaf9786

Add getXHRBodyLength and getFetchBodyLength functions to opentelemetr…

d6149ca

…y-sdk-trace-web

track request body content length in xhr instrumentation

d97b02b

track request body content length in fetch instrumentation

860557e

add changelog entry

9dfe663

update browser; webworker tests

a14c1f9

- webworkers don't have DOMParser or Document APIs - hard-coding an expected length for Document length is fragile (and varies across platforms)

account for older platforms where ReadableStream can't tee()

499c8ca

lint fix: add comment after @ts-expect-error

19890b5

make response body measurement opt-in

bee76c8

MustafaHaddara force-pushed the request-body-size branch from 92da8c9 to bee76c8 Compare August 12, 2024 16:55

update named exports

854cc53

MustafaHaddara commented Aug 23, 2024

View reviewed changes

tbrockman reviewed Aug 24, 2024

View reviewed changes

packages/opentelemetry-sdk-trace-web/src/utils.ts Outdated Show resolved Hide resolved

MustafaHaddara force-pushed the request-body-size branch from cada7e9 to 8c42240 Compare August 26, 2024 13:25

support formData.entries()

d011829

MustafaHaddara force-pushed the request-body-size branch from 8c42240 to d011829 Compare August 26, 2024 14:10

MustafaHaddara added 4 commits August 26, 2024 10:20

update stream in fetch tests

178e73b

add a warning for old platforms

74cb0ea

Merge branch 'main' of github.com:open-telemetry/opentelemetry-js int…

7c2bfe2

…o request-body-size

fix fetch tests

51cc125

MustafaHaddara and others added 2 commits August 27, 2024 12:04

Merge branch 'main' of github.com:open-telemetry/opentelemetry-js int…

f0dcca9

…o request-body-size

Merge branch 'main' into request-body-size

192f440

tbrockman reviewed Sep 6, 2024

View reviewed changes

MSNev approved these changes Oct 15, 2024

View reviewed changes

Merge branch 'main' into request-body-size

0007def

MustafaHaddara requested a review from a team as a code owner October 16, 2024 15:06

martinkuba approved these changes Oct 22, 2024

View reviewed changes

pichlermarc requested a review from a team October 23, 2024 09:53

pichlermarc requested changes Oct 23, 2024

View reviewed changes

MustafaHaddara added 6 commits October 25, 2024 11:47

use semconv 1.7 attribute

40d01bb

update changelog

8192b86

Merge branch 'main' into request-body-size

f73fccd

update changelog

8b7a76d

lint markdown

93957de

use the correct attribute -.-

1c26261

Merge branch 'main' of github.com:open-telemetry/opentelemetry-js int…

8d6705f

…o request-body-size

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track request body size in XHR and Fetch instrumentations #4706

Track request body size in XHR and Fetch instrumentations #4706

MustafaHaddara commented May 14, 2024 •

edited

Loading

codecov bot commented May 22, 2024 •

edited

Loading

MustafaHaddara commented Aug 12, 2024

MustafaHaddara Aug 23, 2024

tbrockman Aug 24, 2024

MustafaHaddara commented Aug 26, 2024

tbrockman Sep 6, 2024 •

edited

Loading

tbrockman Sep 6, 2024 •

edited

Loading

JamieDanielson commented Sep 25, 2024 •

edited

Loading

MustafaHaddara commented Oct 15, 2024

MustafaHaddara commented Oct 16, 2024

pichlermarc Oct 23, 2024

MustafaHaddara Oct 25, 2024

pichlermarc Oct 23, 2024 •

edited

Loading

MustafaHaddara Oct 25, 2024

MustafaHaddara commented Nov 5, 2024

Track request body size in XHR and Fetch instrumentations #4706

Are you sure you want to change the base?

Track request body size in XHR and Fetch instrumentations #4706

Conversation

MustafaHaddara commented May 14, 2024 • edited Loading

Which problem is this PR solving?

Short description of the changes

Detailed Description

Type of change

How Has This Been Tested?

Checklist:

codecov bot commented May 22, 2024 • edited Loading

Codecov Report

MustafaHaddara commented Aug 12, 2024

MustafaHaddara Aug 23, 2024

Choose a reason for hiding this comment

tbrockman Aug 24, 2024

Choose a reason for hiding this comment

MustafaHaddara commented Aug 26, 2024

tbrockman Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

tbrockman Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

JamieDanielson commented Sep 25, 2024 • edited Loading

MustafaHaddara commented Oct 15, 2024

MustafaHaddara commented Oct 16, 2024

pichlermarc Oct 23, 2024

Choose a reason for hiding this comment

MustafaHaddara Oct 25, 2024

Choose a reason for hiding this comment

pichlermarc Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

MustafaHaddara Oct 25, 2024

Choose a reason for hiding this comment

MustafaHaddara commented Nov 5, 2024

MustafaHaddara commented May 14, 2024 •

edited

Loading

codecov bot commented May 22, 2024 •

edited

Loading

tbrockman Sep 6, 2024 •

edited

Loading

tbrockman Sep 6, 2024 •

edited

Loading

JamieDanielson commented Sep 25, 2024 •

edited

Loading

pichlermarc Oct 23, 2024 •

edited

Loading