Adds _doc_count for spans and transactions #109
Conversation
| @@ -81,12 +83,19 @@ func (e *APMEvent) MarshalFastJSON(w *fastjson.Writer) error { | |||
| if e.Transaction != nil { | |||
| e.Transaction.toModelJSON(&transaction, e.Processor != nil && e.Processor.Name == "metric" && e.Processor.Event == "metric") | |||
| doc.Transaction = &transaction | |||
|
|
|||
| if e.Transaction.GetRepresentativeCount() >= 1 { | |||
| doc.DocCount = int64(math.Round(e.Transaction.GetRepresentativeCount())) | |||
There was a problem hiding this comment.
@elastic/apm-server perhaps we can distribute the rounding? e.g. if representative count is 1.2, there's a 80% chance it will be rounded down to 1, and 20% it will be rounded up to 2. Or is that a dumb idea?
There was a problem hiding this comment.
I'd rather keep it simple to start with, and round. Hopefully we get a more explicit means of weighting statistics by a floating point field in the future.
dgieselaar
force-pushed
the
doc-count-repr-count
branch
from
a73e02e
to
300789c
Compare
There was a problem hiding this comment.
Could we do this in the ingest pipeline instead? https://github.com/elastic/apm-server/blob/main/apmpackage/apm/data_stream/traces/elasticsearch/ingest_pipeline/default.yml
That way old data will get the field too. Also it will reduce the wire size slightly.
I guess we don't have a better option at the moment, but the use of _doc_count has always been too magical for my liking. I think ideally Elasticsearch would provide a way to explicitly weight aggregations by another field. You can already do weighed averages -- why not for other metrics aggs too?
|
@axw pipeline sounds fine to me, I'll close this one and move it there. Before I open it, do you have any opinion on distributing the rounding as described in #109 (comment)?
What do you mean with "old data"? Older APM Server versions sending data to a more recent cluster (with the most recent package installed)?
Agreed, and AFAIK |
Exactly
🥳 |
|
Closing in favor of elastic/apm-server#11217 |
We've recently had a few requests for upscaling spans and transactions, either when falling back to sampled events in the UI, or for custom dashboards. Currently, throughput drops, which makes it hard to use custom dashboards. Latency and failure results can be skewed. If we set _doc_count to the value of representative_count, Elasticsearch aggregations will automatically upscale results. This PR makes that change. Not sure if this is the way to do it, but it wasn't much work, so gave it a shot anyway.
See: