Using "active span concept" in asynchronous code can make faulty span connections if not careful

[nic-godz]

I have been using opentelemetry-cpp for tracing & logging for a while now. The metrics part is still to be used(have been testing it)

All C++ applications, using otel-cpp, is asynchronous and based on Boost Asio. My experience is that setting spans to active when you have a lot of asynchronous operations going can can really mess up parent-child span connections.

For instance if an outgoing request is divided into several asynchronous operations and a span is created and set active until reply is received. The scope object is kept as well for the lifetime of the request. Some nested spans are created for DNS resolve, connection, write, ...

But when parallel requests are created the same way, things start to go bad as suddenly other requests starts getting active and child spans will get the wrong connections to parents. Same goes for Logs connected to a specific span context.

My experience is to avoid setting spans to active and instead explicitly set parent span like:
options.parent = parent_span->GetContext();

And if WithActiveSpan(span) is used, the scope should never exist over asynchronous operations. But if you are using Boost Asio with stackful coroutines async code looks like sync code(thats the point) and you can easily forget that execution will leave in the middle of a function and continue later. So if a span scope is a stack variable it will survive over multiple asynchronous operation.

Whats your take on this? I have really no good ideas here. I guess it all boils down to the execution of ::RuntimeContext::GetCurrent() ? But there is really no magic you can use here to solve this. How is this handled in other languages?

[lalitb]

The use of the "active span concept" is not recommended for asynchronous code. This is because the active span relies on thread-local context for its management, but asynchronous code may operate across multiple threads internally. Your best bet would be to pass the parent span id across the async code (by capturing in lambda or other means) and explicitly set it while creating child span.

How is this handled in other languages?

I know languages like Rust, JS, Python, .Net support propagating context across the async/await calls or the futures. C++ doesn't have any such mechanism supported.

[nic-godz]

Thanks for quick response. Yes it sounds like my experience and even if the asynchronous code is not using more then one thread you will run into problem. I guess threaded applications(running synchronous in each thread) can utilize the thread local context in order to pick up the correct active span. But that is not the way to go for an event driven design on Linux.

I will for sure check out Rust and it's capabilities of propagating context in async/wait calls.
Thanks

[lalitb]

I will for sure check out Rust and it's capabilities of propagating context in async/wait calls.

If it helps, I have written the HTTP client/server example code for Otel Rust to demonstrate propagating context to async function - https://github.com/open-telemetry/opentelemetry-rust/tree/main/examples/tracing-http-propagator/src. Basically, the HTTP server main task/thread spawns an async task for every HTTP request received, and it passes the parent span context to that async task.

[nic-godz]

I'm not fluent in Rust but I must say that ".with_context(cx).await" sure looks good and is of course something we miss in C++ & Boost Asio. Really nice exmple.

[nic-godz]

An old thread but lifting this issue(s) again as initially my concern was about active Span and logging. But I can now see the same problem in the Carrier concept as well. Consider the example code:

auto current_ctx = context::RuntimeContext::GetCurrent();
HttpTextMapCarrier<http_client::Headers> carrier;
auto prop = context::propagation::GlobalTextMapPropagator::GetGlobalPropagator();
prop->Inject(carrier, current_ctx);

Together with Boost Beast multiple asynchronous clients using co_await:
https://www.boost.org/doc/libs/master/libs/beast/example/http/client/awaitable/http_client_awaitable.cpp

Injecting traceparent into every boost::beast::http::request<http::string_body>

As I understands it(might be wrong here) the Carrier concept relies on picking up the Context using context::RuntimeContext::GetCurrent(). The take above was to pass around the parent Span object. But can you get the Context object from the Span? As far as I can see you can only get the SpanContext. So if not. Is the idea to pass around the Context object instead of the Span object? Or both? In the above example it would be something like auto current_ctx = context_; if do_session is a method in a class with context_ as a member variable.

But it feels like a forced construction passing around both Span and Context object. It sure would be nice if you can get the Context from your Span object.

@lalitb any thoughts about my concerns regarding the Carrier concept?

[raulbojalil]

Hi @nic-godz , sorry if my messge is off topic, but I'm trying to implement asynchronous code in my project, what did you do?, there's no documentation around the ENABLE_ASYNC_EXPORT flag.

[nic-godz]

Hi @raulbojalil in conan(version 1.14.2) you have the with_async_export_preview option. Default is False. This will set the env var WITH_ASYNC_EXPORT_PREVIEW. I still think it is default OFF even in the latest 1.17 version(not available on conan.io center yet).

So if you are building from source, make sure to -DWITH_ASYNC_EXPORT_PREVIEW=ON

And if you are downloading using Conan, set with_async_export_preview to True

[raulbojalil]

Thanks @nic-godz, got it working now.