[exporter/signalfx] enabled http2 healthcheck #29716
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR enables the HTTP2 health check to workaround the issue described here open-telemetry/opentelemetry-collector#9022
As to why I chose 10 seconds for
HTTP2ReadIdleTimeout
and5 seconds10 seconds (see review comment) forHTTP2PingTimeout
Those values have been tested in production and they will result, in an active env (with default http timeout of 10 seconds and default retry settings), of a single export failure at max before the health check detects the corrupted tcp connection and closes it.
The only drawback is if the connection was not used for over 10 seconds, we might end up sending unnecessary ping frames, which should not be an issue and if it became an issue, then we can tune those settings.
The SFX exporter has multiples http clients:
Link to tracking Issue:
Testing:
signalfx
exportersudo iptables -A OUTPUT -s <source_IP> -p tcp --sport <source_Port> -j DROP
Expected Result:
Actual Result:
tcp_retries2
is configured toDocumentation:
Readme is updated
Disclaimer:
Not all HTTP/2 servers support H2 Ping, however, this should not be a concern as our ingest servers do support H2 ping.
But if you are routing you can check if H2 ping is supported using this script golang/go#60818 (comment)