tracing: replace future names with spawn locations in task spans #3074
Conversation
Signed-off-by: Eliza Weisman <[email protected]>
Signed-off-by: Eliza Weisman <[email protected]>
whoops, i didn't realize i had a vscode plugin installed that does that. Signed-off-by: Eliza Weisman <[email protected]>
Signed-off-by: Eliza Weisman <[email protected]>
Signed-off-by: Eliza Weisman <[email protected]>
| @@ -122,6 +122,7 @@ cfg_rt! { | |||
| /// ```text | |||
| /// error[E0391]: cycle detected when processing `main` | |||
| /// ``` | |||
| #[cfg_attr(tokio_track_caller, track_caller)] | |||
There was a problem hiding this comment.
Note that this enables track_caller on spawn even when it's not used by the "tracing" feature flag. I think this is actually a good thing, since it means the panic when spawn is called outside the runtime will have the callsite's location, which may make debugging a little easier for users?
However, if folks prefer, I can change this to only enable track_caller when the cfg is set and the "tracing" feature flag is enabled.
| target: "tokio::task", | ||
| "task", | ||
| %kind, | ||
| spawn.location = %format_args!("{}:{}:{}", location.file(), location.line(), location.column()), |
There was a problem hiding this comment.
spawn.location feels a little wordy but I couldn't think of something more descriptive off the top of my head. If folks have better ideas, let me know!
There was a problem hiding this comment.
also, we could alternatively record the filename, line number, and column as separate fields on the task span, like
| spawn.location = %format_args!("{}:{}:{}", location.file(), location.line(), location.column()), | |
| spawn.file = %location.file(), | |
| spawn.line = location.line(), | |
| spawn.column = location.column(), |
and rely on the tracing subscriber to format them nicely...but, this results in more verbose, harder to read output when the tracing subscriber that's in use isn't doing anything fancy, so I thought this was nicer.
|
Probably MSRV will be increased in 1.0, so dependency on autocfg should be temporary. However, futures-core may depend on autocfg in the future. (see rust-lang/futures-rs#2207 (comment) and rust-lang/futures-rs#1875 (comment)) |
* master: tracing: replace future names with spawn locations in task spans (tokio-rs#3074) util: add back public poll_read_buf() function (tokio-rs#3079) net: add pid to tokio::net::unix::UCred (tokio-rs#2633) chore: prepare tokio-util v0.5.0 release (tokio-rs#3078)
In a previous PR (#4128), the `spawn.location` field on task spans was structured into 3 separate fields for the `file`, `line`, and `col`. There is a separately created span for blocking tasks which was missed. This caused tasks created with `spawn_blocking` to appear in `tokio-console` without a location, but with an additional "free form" field containing the formatted source code location. This change modifies this span to use the same format. The span creation needs to be separate from the other task spans because it records the function name. This information is useful in the `spawn_blocking` case, but can be "catastrophically long" in the `async fn` case and was removed in #3074.
In a previous PR (#4128), the `spawn.location` field on task spans was structured into 3 separate fields for the `file`, `line`, and `col`. There is a separately created span for blocking tasks which was missed. This caused tasks created with `spawn_blocking` to appear in `tokio-console` without a location, but with an additional "free form" field containing the formatted source code location. This change modifies this span to use the same format. The span creation needs to be separate from the other task spans because it records the function name. This information is useful in the `spawn_blocking` case, but can be "catastrophically long" in the `async fn` case and was removed in #3074.
Motivation
Currently, the per-task
tracingspans generated by tokio'stracingfeature flag include the
std::any::type_nameof the future that wasspawned. When future combinators and/or libraries like Tower are in use,
these future names can get quite long. Furthermore, when formatting
the
tracingspans with their parent spans as context, any other taskspans in the span context where the future was spawned from can also
include extremely long future names.
In some cases, this can result in extremely high memory use just to
store the future names. For example, in Linkerd, when we enable
tokio=traceto enable the task spans, there's a spawned task whosefuture name is 232990 characters long. A proxy with only 14 spawned
tasks generates a task list that's over 690 KB. Enabling task spans
under load results in the process getting OOM killed very quickly.
Solution
This branch removes future type names from the spans generated by
spawn. As a replacement, to allow identifying whichspawncall aspan corresponds to, the task span now contains the source code location
where
spawnwas called, when the compiler supports the#[track_caller]attribute. Sincetrack_callerwas stabilized in Rust1.46.0, and our minimum supported Rust version is 1.45.0, we can't
assume that
#[track_caller]is always available. Instead, we have aRUSTFLAGS cfg,
tokio_track_caller, that guards whether or not we useit. I've also added a
build.rsthat detects the compiler minorversion, and sets the cfg flag automatically if the current compiler
version is >= 1.46. This means users shouldn't have to enable
tokio_track_callermanually.Here's the trace output from the
chatexample, before this change:...and after:
Closes #3073