other and missing bucket support for terms agg #15525

ppisljar · 2017-12-11T16:14:41Z

release note: 'other' and 'missing' bucket for the terms aggregation
resolves #1961

the other bucket will show up if its enabled, and will include everything that is not in top N (size of term agg)
if missing is enabled (label is set) it might show up as a separate bucket (in case its actually one of the top N buckets), if its not it will be included in the other bucket

alexfrancoeur · 2017-12-12T00:17:50Z

@ppisljar looking good! A few comments, observations and open questions below.

I realize this is early, but want to state we probably need a better value than _other_ here. Other is pretty common

Rather than Show other maybe we can provide some additional context here. Group other values or Group values into 'Other' slice. We can always pull @gchaps in for an improved description as well.

Seems like some sort of bug. I chart the top 25 values and only produce one. However, if I check other we see additional values.

I'm not sure if this is a side effect of other or a pie charts normal approach for visualizing too many slices in a small area. It looks like one single white value, maybe there's a better way for handling this.

The other bucket does not seem to update with time

Filtering on a dashboard seems to work well. We'll want to make sure the label here is updated as well.

Open questions:

Is there any need to have this option for splits that are not terms (significant terms, filters, ranges, etc.)?
Should this configuration be on by default for new visualizations?

ppisljar · 2017-12-12T07:52:24Z

thanks @alexfrancoeur

the value ... i used _other_ as Other might actually exist in your dataset (as a real value), i added an option to provide custom label for other bucket
the top 25 produce only one value ('article') but other shows up, because there are probably documents where article is not set (at this point other is 'all the other values + missing values') ... there is another PR up for the missing bucket, which will then split the other bucket into 'all the other values' and 'missing' (two buckets)
yeah that's a pie issue, and i would leave it out of this PR, when we have really small buckets you won't actually see them, it becomes just a white space
time issue ... definitely a bug, looking into it, it should be fixed now
the label on the filter will be the same as in the legend, as mentioned above i added an option so the user can supply a custom label, and we should come up with a good default

my answers to open questions:

for the first stage (this PR) i would limit this to terms only and wait for users to request it on other aggregations. personally i don't see much value in adding it to filters or ranges, as you could always define a filter on your own that includes everything your first N filters don't
i would not turn it on by default as 1. it would break backward compatibility, 2. this might be intensive on es cluster.

gchaps · 2017-12-12T17:11:49Z

@alexfrancoeur @ppisljar I agree with Alex that the UI text "Show Other" should be more specific. Ping me when you're ready to discuss the text. I can also help with the wording for default values.

thomasneirynck

sweeto! this is going to be a yuge addition.

This is an early PR, so some of my comments might be stale (if so, then ignore).

There's two things to evaluate I think:

I would decouple others and missing in two separte checkbox UIs.
I'd do determination of what the actual query has to become up front, in a pre-flight request instead of a postFlight request. It'd preserve the contract of Terms better imho.

thomasneirynck · 2017-12-14T21:13:22Z

src/ui/public/agg_types/controls/order_and_size.html

-      type="number"
-      min="1"
-    >
+  <div class="vis-editor-agg-form-row">


After using these for a while, I'd separate out both options. By combining them it does reduce the footprint a little, but it also makes it less clear what exactly is going to happen.

So I'd just have two parallel options:

A 'show other' checkbox and the other bucket label as one thing we can toggle.

A 'show missing' checkbox and the missing bucket label as one thing we can toggle.

I also think two separate settings would be better. I am not sure if we would still need a checkbox for the "show missing" or if we could just use a textfield for that, labeled something like: "Replace missing values with" and if the field is empty just don't use it, and if it's set to something replace missing values with it. I think we don't need a "user wants to show missing, but don't want to specify a custom label" option for that.

thomasneirynck · 2017-12-14T21:17:34Z

src/ui/public/agg_types/buckets/terms.js

@@ -60,11 +62,33 @@ export function AggTypesBucketsTermsProvider(Private) {
      return agg.getFieldDisplayName() + ': ' + params.order.display;
    },
    createFilter: createFilter,
+    postFlightRequest: async (aggConfigs, aggConfig, searchSourceAggs, resp, nestedSearchSource) => {
+      const filterAgg = buildOtherBucketAgg(aggConfigs, searchSourceAggs, aggConfig, resp);


this code-path should not be hit when the other functionality is turned off

thomasneirynck · 2017-12-14T21:35:03Z

src/ui/public/vis/request_handlers/courier.js

+      }).then(async resp => {
+        for (let i = 0; i < vis.aggs.length; i++) {
+          const agg = vis.aggs[i];
+          if (!agg.type || !agg.type.postFlightRequest) continue;


I would add postFlightRequest to every type but just have it be the identity-function. Then we don't have to do any typechecking here.

I think we should rather be as failsafe as possible and check if the function exists here before calling it, instead of relying on some complete other place always attaching the identity function for us.

thomasneirynck · 2017-12-14T21:37:56Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

@@ -0,0 +1,143 @@
+import _ from 'lodash';


this module seems like an unnecessary abstraction imho, it just scatters the code. I'd add these functions as methods on the Terms bucket agg.

I think since this file alrady contains 179 lines, we shouldn't merge it into the terms class for better readability and there is not much of a harm to make them an own module. Maybe just rename the file to begin with an underscore to make it more clear, this are private helper functions.

thomasneirynck · 2017-12-14T21:49:55Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

+        return;
+      }
+
+      filterAggDsl.filters[key] = {


imho it'd help readability when not creating temp vars. Just access this as resultAgg.filters

thomasneirynck · 2017-12-14T22:00:08Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

+  return [];
+};
+
+const getAggConfigResult = (responseAggs, aggId, bucketKey) => {


I think we have a bug when splitting charts/slices. we get something like:

at getAggConfigResult (http://localhost:5601/prj/bundles/kibana.bundle.js?v=8467:246452:19) at http://localhost:5601/prj/bundles/kibana.bundle.js?v=8467:246460:85 at http://localhost:5601/prj/bundles/commons.bundle.js?v=8467:21259:15 at baseForOwn (http://localhost:5601/prj/bundles/commons.bundle.js?v=8467:20232:14) at http://localhost:5601/prj/bundles/commons.bundle.js?v=8467:21229:18 at Function.<anonymous> (http://localhost:5601/prj/bundles/commons.bundle.js?v=8467:21532:13) at getAggConfigResult (http://localhost:5601/prj/bundles/kibana.bundle.js?v=8467:246459:20) at updateMissingBucket (http://localhost:5601/prj/bundles/kibana.bundle.js?v=8467:246555:28) at Object._callee$ (http://localhost:5601/prj/bundles/kibana.bundle.js?v=8467:246151:17) at tryCatch (http://localhost:5601/prj/bundles/commons.bundle.js?v=8467:81181:40)```

thomasneirynck · 2017-12-14T22:00:44Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

+  _.each(responseAggs, agg => {
+    resultBuckets = [
+      ...resultBuckets,
+      ...getAggConfigResult(agg.aggs, aggId, bucketKey)


we probably don't want to recurse if we don't have agg.aggs. Or put a guard on top. in any case, we probably want a base-case in this recursive function somewhere.

thomasneirynck · 2017-12-14T22:29:43Z

src/ui/public/vis/request_handlers/courier.js

+          const agg = vis.aggs[i];
+          if (!agg.type || !agg.type.postFlightRequest) continue;
+          const nestedSearchSource = new SearchSource().inherits(searchSource);
+          resp = await agg.type.postFlightRequest(vis.aggs, agg, searchSource.get('aggs')(), resp, nestedSearchSource);


I'm not a 100% convinced this postFlightRequest is the correct abstraction.

This only works for terms, where we can conceive of something like "other", where this configuration requires more calls. But this wouldn't transfer to other aggregations, if a postflight makes sense. This also bleeds into courier in kind of a weird way. Also, the toDSL functionality is now broken for Terms. The configuration of terms and all its outputs (dsl, responses, ...) no longer matches the actual result.

Could we flip this? Instead of introducing a postFlightRequest, do preliminary requests at the beginning.

We could do something similar how the geo_hash aggregation rewrites the query in getRequestAggs (https://github.com/thomasneirynck/kibana/blob/05cbb929658590024e231a0791311c8b859e194f/src/ui/public/agg_types/buckets/geo_hash.js#L106-L106). This would mean of course that we make this method (getRequestAggs) async.

If the above doesn't work, we could introduce an async preFlight method that rewrites the query-object. This could do all the intermediate calls required to produce the final query.

In either case, I think it would preserve the contract of the "Terms" configuration better.

i don't think that is possible, we discussed this with @nreese and found few reasons why to do the postFlight instead of preFlight ... i think the main one was that i need the response for this to work

i can not construct correct filters i need for the 'other' bucket construction if i don't have the response from original query back as this could be nested

we are updating the actual response with fake buckets, which can't really happen in pre-flight

The idea would be that you make the original request in the preflight, and collect the other responses in a similar fashion as here, except for the last one. Your last DSL will then be the correct one, matching the terms-config. Right now, we do the first request (not matching the terms config) and stringing together all the subsequent ones in the postflight.

Bargs · 2017-12-15T17:06:23Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

+          const filter = _.cloneDeep(bucket.filter) || currentAgg.createFilter(bucketKey);
+          delete filter.meta;
+          const migratedFilter = migrateFilter(filter.query || filter);
+          const newFilters = [...filters, migratedFilter];


You might want to use the buildQueryFromFilters helper here. It'll migrate and clean (remove meta prop) the filters exactly how we do it in SearchSource and return a valid bool query body that you can use.

Bargs · 2017-12-15T17:10:10Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

+
+      // create not filters for all the buckets
+      const notKeys = agg.buckets.map(bucket => bucket.key);
+      filterAggDsl.filters[key].bool.must_not[0].terms[aggWithOtherBucket.params.field.name] = notKeys;


I don't think you want to use a terms query. If a user is aggregating on an analyzed field (with field data turned on) these term queries won't match properly. Scripted fields also won't work with the current implementation. I think a negated phrases filter would work. It'll create match_phrase queries, so my only question would be whether the terms agg supports any data types that won't work with match_phrase.

It could also make more sense to use the agg's createFilter method, that way you know you're getting a valid filter for this particular agg and field type. It already uses a filter constructor under the hood. Negate them all and then combine them with a bool using the buildQueryFromFilters helper I mentioned above.

(as a general rule, if you ever find yourself manually building query DSL in your code, there's a good chance you should be using one of our filter/query abstractions instead)

alexfrancoeur · 2017-12-16T16:36:19Z

@ppisljar some quick weekend feedback.

I noticed that the behavior of an empty pie chart is a bit odd. Not sure if it's related to the PR. At first I didn't realize I was missing data, just thought that there was an issue loading the pie chart. However, if I resize the window the error message occurs. Check out this gif

A couple of additional comments below.

As a Kibana end user - I don't know the difference between other and missing buckets. If it's necessary to have both I think we need to explicitly state what they are. In other UI's this is normally represented as "Others".

Should a user have a choice to choose a missing and/or other bucket? I understand that elasticsearch uses _other_ as the label, but from a UI perspective it's not that appealing. Using Other by default seems to be a bit more user friendly.

I feel we're running into issues here within the UI that we have in the past, using elasticsearch terminology vs. what would be intuitive for building the visualization. I don't have any good recommendations at the moment but it's something to think about. Maybe we can discuss some options early next week?

ppisljar · 2017-12-18T15:42:09Z

OK so here is what we have at the moment (not part of this PR yet):

we need to come up with:

text next to the checkboxes
text for the input boxes (label for other/missing bucket)
text for the info icons (not present yet) which can give a longer description of what exactly is happening

gchaps · 2017-12-18T18:04:41Z

@ppisljar Let's try this:

Group other values into separate bucket

Label for other bucket

Show missing values

Label for missing values

Question: are the missing & null values in their own bucket or in the "other" bucket

timroes

I tried to break it and haven't found a way so far. Everything seems to work as expected, even if you try to nest different aggregations in weird ways.

I have some more code suggestions and questions, but since I am now away for 2 weeks, these shouldn't block this PR if it would get ready otherwise.

Also I would like to see some more tests and documentation (especially in the terms_other_bucket_helper file), but that's anyway on the todo list.

timroes · 2017-12-22T10:44:31Z

src/ui/public/agg_types/agg_type.js

+     * A function that will be called after the main request has been made
+     * and should return an updated response
+     */
+    this.postFlightRequest = config.postFlightRequest || null;


Add the parameters, that will be passed to the postFlightRequest function to the documentation.

And that the response type is allowed to be async (or a promise).

Is the AggConfig class tied towards the courier request handler, or could this possibly be used with other request handlers? As far as I understand, the AggConfig will always be used when using the default editor with schemas. That doesn't necessarily mean you need the courier request handler though?

If that's the case, I think we should rather implement the postFlightRequest outside of the courier request handler and inside the generic calling the request handler (in visualize.js?). If there are reasons against it, I would at least document here, that the postFlightRequest only works when using the courier request handler.

currently default editor Data tab is tied to the AggConfig and to Courier. I think AggCpnfigs are only used with courier.

timroes · 2017-12-22T10:50:34Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

@@ -0,0 +1,143 @@
+import _ from 'lodash';


I think since this file alrady contains 179 lines, we shouldn't merge it into the terms class for better readability and there is not much of a harm to make them an own module. Maybe just rename the file to begin with an underscore to make it more clear, this are private helper functions.

timroes · 2017-12-22T10:53:32Z

src/ui/public/filter_bar/filter_bar_click_handler.js

@@ -49,6 +49,7 @@ export function FilterBarClickHandlerProvider(Notifier, Private) {
              }
            }
          })
+          .flatten()


So simple and yet so powerful! ✨

timroes · 2017-12-22T10:58:24Z

src/ui/public/vis/request_handlers/courier.js

@@ -26,7 +28,7 @@ const CourierRequestHandlerProvider = function (Private, courier, timefilter) {
      return new Promise((resolve, reject) => {
        if (shouldQuery()) {
          delete vis.reload;
-          searchSource.onResults().then(resp => {
+          searchSource.onResults().then(async resp => {


I think the async here isn't needed?

timroes · 2017-12-22T11:07:29Z

src/ui/public/vis/request_handlers/courier.js

@@ -35,15 +37,22 @@ const CourierRequestHandlerProvider = function (Private, courier, timefilter) {
            };

            searchSource.rawResponse = resp;
-            resolve(resp);
+
+            resolve(_.cloneDeep(resp));


The reason for cloning this, was the spy panel? So this would be redundant with the refactored requests panel? (Doesn't mean that we should remove it here, I just want to make sure I understand it.)

timroes · 2017-12-22T11:16:05Z

src/ui/public/vis/request_handlers/courier.js

-
-
+      }).then(async resp => {
+        for (let i = 0; i < vis.aggs.length; i++) {


We could use for (const agg of vis.aggs) here, since we don't need the index i anywhere inside the loop.

timroes · 2017-12-22T11:28:20Z

src/ui/public/agg_types/buckets/terms.js

+        const filterAgg = buildOtherBucketAgg(aggConfigs, searchSourceAggs, aggConfig, resp);
+        nestedSearchSource.set('aggs', filterAgg);
+        const response = await nestedSearchSource.fetchAsRejectablePromise();
+        // todo: refactor to not have side effects


I think that comment can go away, seems, the function doesn't have any side effects anymore.

timroes · 2017-12-22T13:08:57Z

src/ui/public/agg_types/buckets/terms_other_bucket_helper.js

+ * @param otherAgg: AggConfig of the aggregation with other bucket
+ */
+const getOtherAggTerms = (requestAgg, key, otherAgg) => {
+  return requestAgg['other-filter'].filters.filters[key].bool.must_not.map(filter => {


Maybe this version looks a bit nicer:

return requestAgg['other-filter'].filters.filters[key].bool.must_not.filter(filter => filter.match_phrase && filter.match_phrase[otherAgg.params.field.name] ).map(filter => filter.match_phrase[otherAgg.params.field.name].query );

alexfrancoeur

LGTM with one final comment. While these new options are keyboard accessible, I don't seen a bounding box or indicator that I'm in the checkbox. Would you mind adding the appropriate styling here? We should visually indicate that the box is in focus.

…tRequest function

nreese · 2018-01-05T13:25:38Z

@ppisljar This is great. Nice job.

There is one point of UI confusion. When adding a filter on the Other bucket, the UI does not clearly reflect that the Other filter is a not filter.

The created filters work great and make a lot of sense

ppisljar · 2018-01-05T14:12:47Z

thanks @nreese, above is not relevant to this PR, as the same happens with any NOT filter. please create an issue for it.

ppisljar added Feature:Visualizations Generic visualization features (in case no more specific feature label is available) WIP Work in progress labels Dec 11, 2017

ppisljar requested review from timroes, thomasneirynck and alexfrancoeur December 11, 2017 16:14

ppisljar requested a review from nreese December 12, 2017 16:31

ppisljar requested a review from Bargs December 12, 2017 17:35

ppisljar mentioned this pull request Dec 14, 2017

[WIP] adding missing aggregation and missing option to terms agg #15295

Closed

ppisljar changed the title ~~[WIP] other bucket support for terms agg~~ [WIP] other and missing bucket support for terms agg Dec 14, 2017

thomasneirynck reviewed Dec 14, 2017

View reviewed changes

Bargs reviewed Dec 15, 2017

View reviewed changes

ppisljar force-pushed the enh/otherBucket branch 2 times, most recently from aba94e8 to ae0e053 Compare December 21, 2017 15:55

timroes approved these changes Dec 22, 2017

View reviewed changes

thomasneirynck self-requested a review December 26, 2017 15:48

alexfrancoeur reviewed Dec 27, 2017

View reviewed changes

ppisljar force-pushed the enh/otherBucket branch from 9a11a10 to 70b6a57 Compare January 3, 2018 15:10

ppisljar added release_note:enhancement review v6.2.0 v7.0.0 and removed WIP Work in progress labels Jan 3, 2018

ppisljar changed the title ~~[WIP] other and missing bucket support for terms agg~~ other and missing bucket support for terms agg Jan 3, 2018

ppisljar force-pushed the enh/otherBucket branch from 1a98010 to f4ce5e3 Compare January 4, 2018 07:22

ppisljar added 20 commits January 5, 2018 13:07

fixing nesting under filters agg

33a4baf

fixing some issues with nesting

fd4e38c

some refactoring and improving the way filters work

01ed0ff

moving code out of courier to the terms agg and introducing postFligh…

6e2e060

…tRequest function

cleaning up

c395090

updating filter generation based on @Bargs suggestions

d528993

support for missing bucket

175dd70

improving texts

7de978e

fixing missing bucket error

bab7c67

updating based on review

31b1a5b

updating based on matt's suggestions

f2be0bf

apply exists filter if missingBucket is not enabled

b9bd181

allow bucket to set multiple filters

9147e54

removing side effects from functions

7eb3cfb

fixing tabify response converter

9169b9f

fixing some more issues

9bc1e36

updating based on review

8682c33

adding unit tests

7a6f498

adding selenium test

ac04b00

updating based on review from thomas

faf0eca

ppisljar force-pushed the enh/otherBucket branch from 84c55f0 to faf0eca Compare January 5, 2018 12:34

ppisljar merged commit 2fd41d5 into elastic:master Jan 5, 2018

ppisljar added a commit to ppisljar/kibana that referenced this pull request Jan 5, 2018

other and missing bucket support for terms agg (elastic#15525)

014101f

ppisljar mentioned this pull request Jan 5, 2018

[6.x] other and missing bucket support for terms agg (#15525) #15864

Merged

nreese mentioned this pull request Jan 5, 2018

display NOT when new filter is negated #15865

Merged

ppisljar added a commit that referenced this pull request Jan 8, 2018

other and missing bucket support for terms agg (#15525) (#15864)

acbdf48

gmoskovicz mentioned this pull request Apr 12, 2018

Add Other bucket as an option for Filter Aggregations #17674

Closed

aphelionz mentioned this pull request May 7, 2018

Accessibility: Kibana 6.3 Meta Issue #18866

Closed

48 tasks



		}).then(async resp => {
		for (let i = 0; i < vis.aggs.length; i++) {

other and missing bucket support for terms agg #15525

other and missing bucket support for terms agg #15525

Conversation

ppisljar commented Dec 11, 2017 • edited Loading

alexfrancoeur commented Dec 12, 2017

ppisljar commented Dec 12, 2017 • edited by timroes Loading

gchaps commented Dec 12, 2017

thomasneirynck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasneirynck Dec 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasneirynck Dec 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexfrancoeur commented Dec 16, 2017

ppisljar commented Dec 18, 2017

gchaps commented Dec 18, 2017

timroes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexfrancoeur left a comment

Choose a reason for hiding this comment

nreese commented Jan 5, 2018

ppisljar commented Jan 5, 2018

ppisljar commented Dec 11, 2017 •

edited

Loading

ppisljar commented Dec 12, 2017 •

edited by timroes

Loading

thomasneirynck Dec 14, 2017 •

edited

Loading

thomasneirynck Dec 14, 2017 •

edited

Loading