[data views] More efficient telemetry collection #152298

mattkime · 2023-02-27T22:06:53Z

Summary

Telemetry no longer makes field caps calls. Also batch loads data view saved objects.

To collect telemetry -

POST kbn:/api/telemetry/v2/clusters/_stats
{
}

Addresses: https://github.com/elastic/sdh-kibana/issues/3389

Checklist

Unit or functional tests were updated or added to match the most common scenarios

mattkime · 2023-02-27T22:57:12Z

src/plugins/data_views/server/register_index_pattern_usage_collection.ts

+          results.runtimeFieldCount / results.indexPatternsWithRuntimeFieldCount;
+      }
+    });
+  }

  return results;


console.log here to see results. results are encrypted in the http response. You may need to restart kibana to run this multiple times.

elasticmachine · 2023-02-27T23:15:23Z

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

kertal

Looks like this will be huge improvement in terms efficiency 👍
Tried it with my local setup, I was just testing data views with no indices. it threw an exception

kertal · 2023-02-28T13:57:15Z

src/plugins/data_views/server/register_index_pattern_usage_collection.ts

-      results.indexPatternsWithRuntimeFieldCount++;
-      results.runtimeFieldCount += runtimeFields.length;
+    savedObjects.forEach((obj) => {
+      const fields = JSON.parse(obj.attributes.fields) || [];


the JSON.parse threw an exception locally, so it's recommendable to use atry{}catch(){} pattern for this case

Ah, I had failed to push a change. I also changed the default data view definition in the test not to have the attributes that are used as to provide test coverage.

kertal · 2023-02-28T13:58:03Z

src/plugins/data_views/server/register_index_pattern_usage_collection.ts

+    savedObjects.forEach((obj) => {
+      const fields = JSON.parse(obj.attributes.fields) || [];
+      const runtimeFieldsMap = obj.attributes.runtimeFieldMap
+        ? JSON.parse(obj.attributes.runtimeFieldMap) || {}


This is also a candidate for try - catch

@mattkime Telemetry stats now work as excepted 👍 ... one think I still think worth to consider is using a try - catch pattern for the JSON.parse. Because if for whatever reason an invalid JSON string is passed in, a SyntaxError is thrown, and this would lead and the one hostile data view would stop the generation of the results ... or do I miss somewhere this was caught? ... or would such a case also lead to troubles before this change?

…kime/kibana into data_views_telemetry_more_efficient

jughosta · 2023-02-28T16:26:27Z

src/plugins/data_views/server/register_index_pattern_usage_collection.ts


-    const scriptedFields = ip.getScriptedFields();
-    const runtimeFields = ip.fields.filter((fld) => !!fld.runtimeField);
+  const finder = savedObjectsService.createPointInTimeFinder<DataViewFieldAttrs>(findOptions);


Should we also call finder.close() or it's not required?

Its not needed with the for loop.

rudolf · 2023-03-01T10:32:04Z

It'll be easier if we could get that merged into main, but would be good to test this with #151110

kertal

Tested it and I found that when using our demo data views, the result of the telemetry is different

Before:

{
indexPatternsCount: 3,
indexPatternsWithScriptedFieldCount: 0,
indexPatternsWithRuntimeFieldCount: 2,
scriptedFieldCount: 0,
runtimeFieldCount: 2,
perIndexPattern: {
scriptedFieldCount: { min: undefined, max: undefined, avg: undefined },
runtimeFieldCount: { min: 1, max: 1, avg: 1 },
scriptedFieldLineCount: { min: undefined, max: undefined, avg: undefined },
runtimeFieldLineCount: { min: 1, max: 1, avg: 1 }
}
}

After

{
indexPatternsCount: 3,
indexPatternsWithScriptedFieldCount: 0,
indexPatternsWithRuntimeFieldCount: 0,
scriptedFieldCount: 0,
runtimeFieldCount: 0,
perIndexPattern: {
scriptedFieldCount: { min: undefined, max: undefined, avg: undefined },
runtimeFieldCount: { min: undefined, max: undefined, avg: undefined },
scriptedFieldLineCount: { min: undefined, max: undefined, avg: undefined },
runtimeFieldLineCount: { min: undefined, max: undefined, avg: undefined }
}
}

so the field count doesn't seem to work

…kime/kibana into data_views_telemetry_more_efficient

mattkime · 2023-03-01T20:38:00Z

@kertal fixed. I was requesting the fields wrong, its 'runtimeFieldMap' not 'attributes.runtimeFieldMap'

kibana-ci · 2023-03-19T04:38:55Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 8142f67

Failed CI Steps

FTR Configs #18

Test Failures

[job] [logs] FTR Configs #18 / security APIs - Session Idle Session POST /internal/security/session should extend the session

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	433	436	+3

References to deprecated APIs

id	before	after	diff
`dataViews`	150	148	-2

Total ESLint disabled count

id	before	after	diff
`securitySolution`	513	516	+3

History

💔 Build #111914 failed c8c8f07
💚 Build #111588 succeeded 30af38c
💛 Build #111510 was flaky e025579
💚 Build #111274 succeeded cfa68df
💔 Build #111157 failed e575341

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mattkime

mattkime · 2023-03-19T22:44:46Z

@kertal I think I still need an official 👍

kertal

LGTM, tested the payload sent for telemetry before and with this PR using our demo datasets. it's identical 👍

## Summary Telemetry no longer makes field caps calls. Also batch loads data view saved objects. To collect telemetry - ``` POST kbn:/api/telemetry/v2/clusters/_stats { } ``` Addresses: elastic/sdh-kibana#3389 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 1892bf6)

kibanamachine · 2023-03-20T12:57:42Z

💚 All backports created successfully

Status	Branch	Result
✅	8.7

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…3293) # Backport This will backport the following commits from `main` to `8.7`: - [[data views] More efficient telemetry collection (#152298)](#152298)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Matthew Kime <[email protected]>

## Summary Telemetry no longer makes field caps calls. Also batch loads data view saved objects. To collect telemetry - ``` POST kbn:/api/telemetry/v2/clusters/_stats { } ``` Addresses: elastic/sdh-kibana#3389 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

more efficient data view telemetry

03457cb

mattkime changed the title ~~more efficient data view telemetry~~ [data views] More efficient telemetry collection Feb 27, 2023

mattkime commented Feb 27, 2023

View reviewed changes

mattkime marked this pull request as ready for review February 27, 2023 23:15

mattkime requested a review from a team as a code owner February 27, 2023 23:15

Merge branch 'main' into data_views_telemetry_more_efficient

dc1c978

kertal assigned mattkime Feb 28, 2023

kertal reviewed Feb 28, 2023

View reviewed changes

check for fields attribute

aa5b559

kertal reviewed Feb 28, 2023

View reviewed changes

mattkime added 3 commits February 28, 2023 07:59

test without fields array and runtime field map

63d8836

Merge branch 'data_views_telemetry_more_efficient' of github.com:matt…

9cb6a5f

…kime/kibana into data_views_telemetry_more_efficient

remove comment

bcc4ab8

jughosta reviewed Feb 28, 2023

View reviewed changes

mattkime added 2 commits February 28, 2023 11:44

Merge branch 'main' into data_views_telemetry_more_efficient

e575341

Merge branch 'main' into data_views_telemetry_more_efficient

cfa68df

mattkime requested a review from kertal March 1, 2023 11:32

kertal reviewed Mar 1, 2023

View reviewed changes

mattkime added 3 commits March 1, 2023 13:37

Merge branch 'main' into data_views_telemetry_more_efficient

eea5c03

fix attribute references

518aea8

Merge branch 'data_views_telemetry_more_efficient' of github.com:matt…

e025579

…kime/kibana into data_views_telemetry_more_efficient

Merge branch 'main' into data_views_telemetry_more_efficient

30af38c

kertal self-requested a review March 2, 2023 11:40

mattkime added 2 commits March 4, 2023 20:05

Merge branch 'main' into data_views_telemetry_more_efficient

c8c8f07

Merge branch 'main' into data_views_telemetry_more_efficient

8142f67

kertal approved these changes Mar 20, 2023

View reviewed changes

mattkime merged commit 1892bf6 into elastic:main Mar 20, 2023

kibanamachine added the v8.8.0 label Mar 20, 2023

kibanamachine mentioned this pull request Mar 20, 2023

[8.7] [data views] More efficient telemetry collection (#152298) #153293

Merged

kibanamachine added the v8.7.0 label Mar 20, 2023

mattkime mentioned this pull request Mar 23, 2023

Data views usage collection causes large amount of Elasticsearch queries #151064

Closed

kertal added the performance label Apr 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[data views] More efficient telemetry collection #152298

[data views] More efficient telemetry collection #152298

mattkime commented Feb 27, 2023 •

edited by kibanamachine

Loading

mattkime Feb 27, 2023

elasticmachine commented Feb 27, 2023

kertal left a comment

kertal Feb 28, 2023

mattkime Feb 28, 2023

kertal Feb 28, 2023

kertal Mar 2, 2023

jughosta Feb 28, 2023

mattkime Feb 28, 2023

rudolf commented Mar 1, 2023

kertal left a comment •

edited

Loading

mattkime commented Mar 1, 2023

kibana-ci commented Mar 19, 2023

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

mattkime commented Mar 19, 2023

kertal left a comment

kibanamachine commented Mar 20, 2023

[data views] More efficient telemetry collection #152298

[data views] More efficient telemetry collection #152298

Conversation

mattkime commented Feb 27, 2023 • edited by kibanamachine Loading

Summary

Checklist

mattkime Feb 27, 2023

Choose a reason for hiding this comment

elasticmachine commented Feb 27, 2023

kertal left a comment

Choose a reason for hiding this comment

kertal Feb 28, 2023

Choose a reason for hiding this comment

mattkime Feb 28, 2023

Choose a reason for hiding this comment

kertal Feb 28, 2023

Choose a reason for hiding this comment

kertal Mar 2, 2023

Choose a reason for hiding this comment

jughosta Feb 28, 2023

Choose a reason for hiding this comment

mattkime Feb 28, 2023

Choose a reason for hiding this comment

rudolf commented Mar 1, 2023

kertal left a comment • edited Loading

Choose a reason for hiding this comment

mattkime commented Mar 1, 2023

kibana-ci commented Mar 19, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

History

mattkime commented Mar 19, 2023

kertal left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 20, 2023

💚 All backports created successfully

Questions ?

mattkime commented Feb 27, 2023 •

edited by kibanamachine

Loading

kertal left a comment •

edited

Loading