How to set OpenTelemetry tracing parameters?

[boramalper]

I am using Knative Eventing and want to set OpenTelemetry tracing parameters (tracing ID, span ID etc.) for the event that I am sending to Knative Eventing Broker.

I thought WithTracePropagation() could have been helpful but it is deprecated and no alternatives are listed. ObservabilityService could have been an alternative solution but it does not have an OutboundContextDecorators()... Though I would like to be able to set OpenTelemetry tracing parameters myself freely, without using middlewares or other magic.

What I currently have is this (simplified):

import(
	cloudevents "github.com/cloudevents/sdk-go/v2"
	"github.com/google/uuid"
)


type MyStruct struct {
	// ...
}

func main() {	
	event := cloudevents.NewEvent("1.0")
	event.SetID(uuid.New().String())
	event.SetType("my-type")
	event.SetSource("//my-source")

	_ = event.SetData(cloudevents.ApplicationJSON, MyStruct{})

	ceClient, _ := cloudevents.NewClientHTTP()
	// I want to set OpenTelemetry trace ID here!
	if result := ceClient.Send(cloudevents.ContextWithTarget(ctx, env.Sink), event); !cloudevents.IsACK(result) {
		log.Fatalf("failed to send CloudEvent: %+v", result)
	}
}

[joaopgrassi]

Hey, @boramalper are still looking for an answer to this? I recently did an implementation of the Observability service, working with OpenTelemetry. If I understood correctly you want to put the trace data (traceparent) inside the event before you send it? The approach I took is to the use DistributedTracingExtension, in conjunction with a custom carrier from OTel.

You can check it here: https://github.com/dynatrace-oss-contrib/go-utils/blob/feat/2422/add-otel-instrumentation/pkg/common/observability/otel_observability_service.go, here: https://github.com/dynatrace-oss-contrib/go-utils/blob/feat/2422/add-otel-instrumentation/pkg/common/observability/cloudevents_carrrier.go and here: https://github.com/dynatrace-oss-contrib/go-utils/blob/feat/2422/add-otel-instrumentation/pkg/common/observability/cloudevents_trace_context.go

Then when sending the event you can do:

func sendEvent(ctx context.Context, e cloudevents.Event) {

    // will inject the traceparent from the ctx into the event
    // Or you could also start a span here and use the new context
    
    // ctx, span := tracer.Start(ctx, "sending", trace.WithSpanKind(trace.SpanKindProducer))
	// defer span.End()
    InjectDistributedTracingExtension(ctx, e)
    
    // send the event..
}

Then on the other side receiving, you can extract it to continue the trace:

func longRunningReceive() {

    evt := getFromQueue()

    // ctx here will have again the proper tracecontext and you can pass it around to continue the trace
    ctx := ExtractDistributedTracingExtension(context.Background(), *evt)
}

Not sure that helps? I might contribute this to cloudevents, just need some discussion on the details/API.

[boramalper]

@ustiugov

[ustiugov]

hi @joaopgrassi, this is definitely something we would be interested in. Would be great if you could upstream that.

thanks @boramalper for bringing me into this discussion.

[joaopgrassi]

Hi @ustiugov cool, I'll work on it. I feel though there's some scattered issues about this. I found this old one #554 and commented there about the sdk spec about this observability service. Maybe we can consolidate things?

[n3wscott]

Yes please, glad you got it working. @benmoss was also helping clear some of this up with some examples added to the sdk. Maybe Ben has more to say?

[benmoss]

With the changes I made in #708 you can propagate trace headers through with the otelhttp middleware, I did this in this demo app: https://github.com/benmoss/knative-tracing/blob/main/cmd/coinflip/main.go#L60-L62

I think this is nice from a simplicity perspective, but having an ObservabilityService implementation seems like a good idea too.

[ustiugov]

@benmoss thanks a lot for your response! this seems very useful and easy to use. do you know what is the minimal version of otelhttp that supports it?

[ustiugov]

@n3wscott thanks a lot for completing the migration to OpenTelemetry, we are definitely going to try it out in vHive in future (not sure when though) and report back.

[benmoss]

@ustiugov it looks like it's been in since 0.12.0 https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]