Pipeline aggregations

[tarunsapra]

It would be nice to have Kibana 4.2 beta support pipeline aggregations of ES which is expected in ES 2.0 beta.

[leoatavalancha]

+1

[Pigueiras]

+1

[kazeborja]

+1

[pemontto]

👍

[rcrezende]

+1

[novozhenets]

+1

[webmstr]

+1

[AlexIoannides]

+1

If I could compute a max over a set of maxes, that would make me very happy.

[joelclimbsthings]

Emphatic +1

[francoisprunier]

+1. That would definitely be a big selling point for our customers, it happens very often that they want a ratio of some kind (request success rate for a basic example) included in their dashboard, and we can't do it at the moment.

[matwolb]

+1. This would be really useful!

[leoatavalancha]

ES 2.0 beta was released a few days ago. Would be really nice to have this feature implemented on Kibana. Ratios are highly frequently used on any dashboard.

[davmrtl]

+1

[thehybridtechnician]

+1

[shendaras]

+1

[fpbouchard]

+1

[amitripshtos]

+1

[skundrik]

+10, this would be great for performance visualisations (per second type like max count/s over 10 minutes) without having to pull all the seconds buckets to client for processing.

[tcucchietti]

+1

[hrchu]

+1

[l8liu]

It will be great that if Kibana could support the visualization of the moving average of the aggregated time series counts (e.g., daily count) and control chart as shown in this blog

https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-1

It is really important for the application of real time surveillance and monitoring system.

[oryband]

👍 @ofir-petrushka @bolshoy @doodyparizada @fuzzyami @avivl @gadisr

[avivl]

💯

[bolshoy]

+1

[jmaynier]

+1

[id3a]

Does anyone know if the Devs are working on this issues? This will be a important feature for kibana especially for dashboards with SLA's and KPI's

[simianhacker]

I have a PR #9725 slated for 5.4 (fingers crossed) that will add full support for pipeline aggregations. It mainly focuses on the time series use case and introduces 5 compatible visualizations.

[simianhacker]

There is also some work being done to add derivative support to the current Kibana vis builder for histograms.

[id3a]

@simianhacker what about a data table visualization with simple pipeline aggregations.
something similar to this:
https://discuss.elastic.co/t/workaround-for-lack-of-support-of-pipeline-aggregations/73551/2

[simianhacker]

The markdown vis is enhanced with Handlebars.js and you have full access to the data through embedded variables. It also supports some HTML so putting together tables is a pretty easy task.

[simianhacker]

@id3a Here is what it looks like to create tables with data (with the Markdown vis). The "overall sum" is a pipeline (sibling) aggregation.

The user has access to ALL the series returned based on what they setup in the "data" tab. The "Markdown" visualization is kind of a Frankenstein version of "Common Mark Markdown" and "Handlebars Templates"

The user can also add custom CSS styling to the Markdown...

As you can see, it's pretty powerful and allows you to create your own custom tables (or whatever 😄 )

[leemengtw]

+1, looking forward to pipeline aggregation visualization!

[id3a]

@simianhacker That would be a great feature to add as a new visualization.
Can you provide a zip file with this plugin?

[simianhacker]

@id3a I can't but my branch is here: https://github.com/simianhacker/kibana/tree/metrics

I'm actively working with the visualize team to review this PR and have it merge as a new feature; probably going to be marked as experimental and all that it entails. Here are a few more examples of pipelines in action:

[AndyRubio]

+1

[tbragin]

via @oras from #10306

I have some results of metrics pass/failed duration.
It will be nice if I could count those results and divide with other results or other field
and display the results in visualization.
The advantages of such of feature, that the results will be changed viewing on the dashboard
order to filter used (Such show only pass results, or show only kind of metrics etc,
of course I can use counter with Logstash of pass/failed but where is all the fun :) of dynamics.
I think it can be implemented using Input of java script/painless or whatever such JSON input
with creating of inner filled in the visualization for display of use.
Those operation semiyearly to Timelion operation just with more options of display
in different graphs rather then time line chart,

[M2UNO3]

+1

[k2xl]

Is this still slated for 5.4?

[epixa]

@k2xl Yep!

[DavidCroenne]

Great demo during Elastic 2017 keynote:
https://www.elastic.co/elasticon/conf/2017/sf/opening-keynote from 30'
Many thanks, Chris!

[k2xl]

Really great job with the demo. Some quick questions

• Super powerful stuff. It wasn't clear if one can adjust parameters of some of the pipeline aggregations like moving averages (changing to HW moving average or to some other type) I might have simply not caught it in the demo.
• Grouping by things like term filter to have multiple line charts - will this possible?
• Can you squeeze this in to 5.3 as a "beta" feature :-D?

Really amazing work!!

[jeffsaremi]

Need a minimal way to use existing pipeline aggs in Kibana. Do you have this in any shape or form? This would be of a bigger priority than allowing UI-based pipeline formulations.
I just need to paste my existing json and show the results in Kibana. thanks

[thomasneirynck]

@jeffsaremi on the master and 5.x branches of Kibana you'll find support for most parent and sibling aggregations in Visualize. Both as part of the supported metrics for line, bar, and area charts, but also as part of the newly introduced Time Series Visual Builder. If you're just using the defaults from ES, without any custom scripting, that might be sufficient for you.

This will be part of the upcoming Kibana 5.4 release.

[jeffsaremi]

@thomasneirynck as far as I know, I cannot do the following visually. However my experience is based on 5.3 release. I have not used the development branch yet. I am not even asking for the visual builder to have this but rather for Kibana to allow such underlying query to be used and its results be shown as visualizations. I think this is a more accomplisahble goal for now.

Please take a look at this. This is a pipeline aggregation to do a sum over a series of max's.

http://stackoverflow.com/questions/43690093/how-to-do-nested-aggregation-in-elasticsearch-where-aggregation-is-over-results/43691556#43691556

[tbragin]

Closing based on functionality shipped in 5.4:

• Visualize: derivative, cumulative sum, moving avg, serial diff; and Sum, Avg, Min and Max bucket pipeline aggregations
  Sum, Avg, Min and Max bucket pipeline aggregation #10070
  cumulative sum, moving avg and serial diff pipeline aggregation metric #10033
  derivative metric aggregation #10020
• Time-series visual builder: support for all pipeline aggs
  Time Series Metric Visualizations  #9725

See the release blog for more info: https://www.elastic.co/blog/kibana-5-4-0-released

[jeffsaremi]

Is there any tutorial for the new aggregations (especially the pipeline ones) and how to use them in Kibana? thanks

[simianhacker]

@jeffsaremi I'm currently working on some video walkthrough (tutorials) for Time Series Visual Builder to be published soon-ish so keep your eye out on the Elastic Blog.

[k2xl]

Congratulations on the release!!!

[HardCoreCodin]

The 'Calculate' feature can only do a bucket-script as a top-level, it can't be applied within a parent bucket-aggregation... :(

And that's only in the time-series visualization, there's no bucket-script in the standard visualizations.

Our particular use-cases require both, and it seems like none are supported - not even in 6.x(rc)
Are there any plans for this?

In our use-cases, each document has a start-time and end-time date-type fields, and we need to generate an overall-duration across buckets of those, grouped on some category-field.
This involves first bucketing on that category, then within that doing 2 metrics: min(start-time) and nax(end-time) of the documents in each bucket, then applying a pipeline-script-bucket agg that uses these 2 metric-aggs to compute the duration between them (max_end_time - min_start_time). That gives us a 'duration' metric as the output of each bucket. We then want to apply some metric-pipeline-aggs on them.

An almost identical structure has been detailed here: https://discuss.elastic.co/t/kibana-5-4-bucket-script-visualization/86475

I was able to express that in the query-DSL, and we're using it in production already - but we can't visualize these in Kibana - which REALLY sucks, as our clients have been requesting such visualizations for years, and we've always told them 'we need support from later versions, for pipeline-aggregations'...
Well, now we have those - but actually, don't...
In time-series builder, we have bucket-script pipeline aggregation that can be expressed, but it can't consume/be-applied-on a parent bucket-term-aggregation, and in the other visualization, we have bucket-term-aggregations, and some pipeline-metric-aggregations, but not the needed bucket-script pipeline aggregation needed to express that...

Is there a workaround using some "advanced JSON" thing I can put my query-DSL in there?

[thomasneirynck]

hi @ArnonMarcus

thanks for expanding on your use-case.

For using the raw query DSL in Kibana to produce a Visualization, you may want to look at the Vega-plugin. https://github.com/nyurik/kibana-vega-vis. It takes a raw query-DSL

This is currently a community contribution, but we are considering bringing it (or something like it) into Kibana: #14911.

Thanks,