Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This library may be enforcing size restrictions that no longer apply #236

Open
kentquirk opened this issue Jan 16, 2024 · 1 comment
Open

Comments

@kentquirk
Copy link
Contributor

Steps to reproduce

  • Periodically, customers have reported that they run into issues where certain OTel spans sent to Refinery are dropped with the message "event exceeds max event size of 100000 bytes, API will not accept this event".
  • Some have also reported they do not get this message when they send the spans directly to Honeycomb via the OTel endpoint.
  • This message comes from libhoney in transmission.go, and the 100K limit is enforced there even before Refinery's libhoney transmission tries to send the data to Honeycomb.
  • The thing is, we believe this limit no longer exists in Honeycomb's ingest. It definitely doesn't exist with OTel ingest, and it doesn't seem to exist for the standard honeycomb API either (/1/events and /1/batch). (More research and testing needed to confirm.)

We should consider whether we can remove this limit from libhoney (and therefore from Refinery).

Related, but not identical, is that there code in Husky that trims fields to 64K but assumes that it doesn't need to do it twice because the span will be limited to 100K.

@MikeGoldsmith
Copy link
Contributor

Next steps:

  • Get the actual field key, value total batch sizes applied at HNY API-
    • Test sending events manually with limits above libhoney's limitations
  • Compare to those imposed here and adjust as necessary
  • Update libhoney to use newer realised limitations
    • This could also be to not try to be smart and apply limitations and just allow the API to reject events

@MikeGoldsmith MikeGoldsmith added status: oncall Flagged for awareness from Honeycomb Telemetry Oncall and removed status: oncall Flagged for awareness from Honeycomb Telemetry Oncall labels Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants