Cannot use a secure OTLP endpoint

[robbkidd]

Describe the bug

I receive a context deadline exceeded error when the agent is configured to send to a TLS OTLP endpoint.

Environment

• OS: Linux (Ubuntu 22.04.2 LTS)
  • uname: Linux 5.19.0-1023-aws #24~22.04.1-Ubuntu SMP Wed Mar 29 15:23:31 UTC 2023 x86_64 GNU/Linux
• Go Version: go version go1.18.1 linux/amd64
• Version: 83cae24

To Reproduce

Steps to reproduce the behavior:

1. make build
2. run:

# sudo isn't needed to repro, but to rule out a permissions issue
sudo \
  OTEL_TARGET_EXE=/usr/bin/ls \
  OTEL_SERVICE_NAME=context-deadline-sadness \
  OTEL_EXPORTER_OTLP_ENDPOINT=api.honeycomb.io:443 \
  ./otel-go-instrumentation

3. Behold! The Error! A deadline has not been met!

Console Output (JSON prettified)

{
	"level": "info",
	"ts": 1681932098.3001456,
	"caller": "cli/main.go:37",
	"msg": "starting Go OpenTelemetry Agent ..."
}
{
	"level": "info",
	"ts": 1681932098.3007176,
	"caller": "opentelemetry/controller.go:107",
	"msg": "Establishing connection to OpenTelemetry collector ..."
}
{
	"level": "error",
	"ts": 1681932108.3095894,
	"caller": "opentelemetry/controller.go:113",
	"msg": "unable to connect to OpenTelemetry collector",
	"addr": "api.honeycomb.io:443",
	"error": "context deadline exceeded",
	"stacktrace": "github.com/open-telemetry/opentelemetry-go-instrumentation/pkg/opentelemetry.NewController\n\t/home/ubuntu/opentelemetry-go-instrumentation/pkg/opentelemetry/controller.go:113\nmain.main\n\t/home/ubuntu/opentelemetry-go-instrumentation/cli/main.go:45\nruntime.main\n\t/usr/lib/go-1.18/src/runtime/proc.go:250"
}
{
	"level": "error",
	"ts": 1681932108.3097224,
	"caller": "cli/main.go:47",
	"msg": "unable to create OpenTelemetry controller",
	"error": "context deadline exceeded",
	"stacktrace": "main.main\n\t/home/ubuntu/opentelemetry-go-instrumentation/cli/main.go:47\nruntime.main\n\t/usr/lib/go-1.18/src/runtime/proc.go:250"
}

Expected behavior

I expect a secure connection to be established. I can make it happen by changing the current hard-coded insecure gRPC config to a hard-coded TLS config:

- conn, err := grpc.DialContext(timeoutContext, endpoint, grpc.WithInsecure(), grpc.WithBlock())
+ conn, err := grpc.DialContext(timeoutContext, endpoint, grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})), grpc.WithBlock())

I imagine the proper fix will be some shenanigans for supporting more of the OTEL_EXPORTER_OTLP_* environment variable config surface area.

[robbkidd]

I think I might have accidentally-on-purpose made a fix for this and most/all of the OTEL_EXPORTER_OTLP_* environment variable while trying to add an "OTel-Go-Auto-Instrumentation" user-agent to the OTLP exporter connection. … which is an issue I meant to open, but could just open a PR for. 😁