feat(*): Implement the skeleton of an OTEL observability system

[calebschoepp]

Summary

This PR introduces OTEL observability into Spin — specifically tracing. Using the nomenclature of #2293 I have added some trigger observability while also cleaning up a bit of the runtime observability. This PR DOES NOT provide full tracing across the entirety of the Spin codebase. That would make for a huge PR so I opted to only trace a few parts and full tracing can be added in a followup PR.

The goal of this PR is to land in a place we're happy with the DX around observability leaving room for us to improve it in future PRs.

Testing

# Install the otel-cli for easily producing distributed traces
go install github.com/equinix-labs/otel-cli@latest

# Run Jaeger
docker run -d -p16686:16686 -p4317:4317 -p4318:4318 -e COLLECTOR_OTLP_ENABLED=true jaegertracing/all-in-one:latest

# Run the app
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 spin up

# Call the app with the otel-cli
OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 otel-cli exec -- bash -c 'curl localhost:3000-v -H "traceparent: $TRACEPARENT"'

[calebschoepp]

@lann any ideas on how to test this feature?

@lann what are the steps to landing this. Any other stakeholders I should pull in for review other than you?

[lann]

@lann any ideas on how to test this feature?

It would probably be fine to do most of the e2e testing in spinkube since most users of telemetry will be there in the medium term. For Spin itself I would probably just test that things don't entirely blow up when telemetry is 1) disabled, 2) enabled without debug traces, 3) enabled with debug traces.

[lann]

@lann what are the steps to landing this. Any other stakeholders I should pull in for review other than you?

I would suggest landing this asap after basic "things don't blow up" tests are in place. We'll want to iterate on user feedback.

[calebschoepp]

@lann any ideas on how to test this feature?

It would probably be fine to do most of the e2e testing in spinkube since most users of telemetry will be there in the medium term. For Spin itself I would probably just test that things don't entirely blow up when telemetry is 1) disabled, 2) enabled without debug traces, 3) enabled with debug traces.

I'll obviously test those cases in Spin locally. But, I'm wondering if we have any sort of e2e testing mechanism in Spin that I could use to automate the verification of it not blowing up.

[rylev]

@calebschoepp We have integration tests that you might be able to cargo cult to fit your needs.

[lann]

@calebschoepp We have integration tests that you might be able to cargo cult to fit your needs.

Yeah you could even run jaeger as a testing-framework service and smoke test the actual span delivery.

[calebschoepp]

@lann @rylev @itowlson this PR is in a place where I'm happy with it and it is ready for a final review.

Once this is approved I'll clean up the commit history.

[rylev]

Looks great!

[itowlson]

90% of my comments are nits or sylistic fretting, and are non-blockers. The one thing I'm concerned about is the error/exit behaviour in main. If I've misread that (which is entirely likely!), let me know and I'll change to an approve!

Thank you for labouring through this. It's a huge plumbing task, but super useful.

[itowlson]

Yay!