-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enabling shared pipeline for profiling by default #1181
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few nits otherwise LGTM
Co-authored-by: Dmitrii Anoshin <[email protected]>
# Logs + Profiling | ||
splunk_hec: | ||
token: "${SPLUNK_HEC_TOKEN}" | ||
endpoint: "${SPLUNK_HEC_URL}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these env vars aren't documented in the deployment instructions, and if they are now intended to be used for profiling we probably want to have a templated endpoint like we do w/ the sapm exporter and use SPLUNK_ACCESS_TOKEN
*. imo we should have a dedicated splunk_hec/profiling
exporter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these env vars aren't documented in the deployment instructions, and if they are now intended to be used for profiling we probably want to have a templated endpoint like we do w/ the sapm exporter and use SPLUNK_ACCESS_TOKEN*
We compose these env vars from SPLUNK_ACCESS_TOKEN
and SPLUNK_REALM
by default. Using these env vars leaves an option for users to send logs to another Splunk endpoint instead of o11y without providing a custom config. Also it's consistent with other configs that we have
imo we should have a dedicated splunk_hec/profiling exporter.
The profiling data shares the same pipeline including otlp receiver. Profiling libraries use the same OTLP port by default. So we cannot split it and create another exporter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the autosetting pass is ok where users aren't setting these env vars for Splunk Cloud* usage. Otherwise wouldn't profiling data be sent directly to Splunk Cloud? (APM/Profiling + Splunk Cloud instead of LO)** Not sure how common this would be.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise wouldn't profiling data be sent directly to Splunk Cloud?
Yes they would, and Splunk Cloud backend can digest and show them is some way, not as good as Splunk O11y tho.
Use case when user wants logs flowing to Splunk Cloud and Profiling data to Splunk O11y will require separate pipelines and changing logs and profiling libraries to send to different ports. So it using SPLUNK_ACCESS_TOKEN
instead of SPLUNK_HEC_TOKEN
doesn't help.
I think we should use SPLUNK_HEC_TOKEN
at least for consistency with other default configs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rmfitzpatrick please let us know if you're ok to merge it
Since the pipeline for logs and profiling is the same, this is a PR to enable it for ec2/ecs and fargate by default to avoid any configuration changes to do so.
Context: https://docs.google.com/document/d/1KABqbnRCi--JGc6ZHIADMeyccTlTrUJZnPVQZk7F4Cs/edit?disco=AAAATAk4C5Y