Improve performance of the Logstash Pipeline Viewer #31293
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ycombinator
added
WIP
|
Pinging @elastic/stack-monitoring |
ycombinator
added
bug
💔 Build Failed |
💔 Build Failed |
|
Hey, nice to see Progress on the Monitoring Features. Will there possibly be a backport to the 6.X branch of Kibana? As with most features in the Monitoring Feature, we need a fix sooner than later. |
ycombinator
force-pushed
the
monitoring-ls-pv-perf
branch
from
a970eaf
to
b9af9e9
Compare
💔 Build Failed |
ycombinator
changed the title
💔 Build Failed |
ycombinator
force-pushed
the
monitoring-ls-pv-perf
branch
from
7abc2ba
to
810b8ab
Compare
💔 Build Failed |
💔 Build Failed |
💔 Build Failed |
|
@chrisronline Quick question about this PR. I'm getting pretty close to wrapping up the changes made by this PR and hope to have them done today. At that point, what will be left is to update the tests. Given the size of this PR, are you okay with me adding the test updates to this PR? Alternatively, I could re-target this PR to a feature branch and then make a follow up PR to the same branch with just test updates, and then finally make a PR from the feature branch to master for everything (which shouldn't need thorough review, just green CI). |
|
@ycombinator Let's do the feature branch approach |
ycombinator
changed the base branch from
master
to
feature/monitoring-ls-pv-perf
💔 Build Failed |
ycombinator
changed the title
ycombinator
changed the base branch from
feature/monitoring-ls-pv-perf
to
master
ycombinator
changed the base branch from
master
to
feature/monitoring-ls-pv-perf
|
@chrisronline I know you already gave your LGTM but that was before I'd addressed your review feedback. I've addressed it now, in case you want to take another look. |
💔 Build Failed |
|
@ycombinator These test failures are directly related to this change right?
|
|
@chrisronline Yes, looking at the failures in CI, that does appear to be the case. [EDIT] There is one failure that is unrelated, however: https://kibana-ci.elastic.co/job/elastic+kibana+pull-request/6090/JOB=x-pack-intake,node=immutable/testReport/junit/X-Pack%20Mocha%20Tests/x-pack_plugins_monitoring_server_lib_logstash___tests___get_pipeline%C2%B7js/get_pipeline__enrichStateWithStatsAggregation_function_enriches_the_state_document_correctly_with_stats/ |
|
@ycombinator what's the desired behavior when a cluster isn't found? If the URL has a non-existent UUID, the server will log: Would it be useful to add some handling to catch the case where the version object, or other related objects are undefined, and return a 404? |
|
I agree if we wanted to fix it it should happen in a subsequent change.
👍 your assessment is fine by me, thanks! |
chrisronline
approved these changes
Mar 21, 2019
CI won't go green on this PR as tests will fail. Per our earlier discussion, we agreed to defer tests to a separate PR. So, two options:
Can you clarify which way you want to proceed? |
|
I misunderstood. Let’s continue with option 2 as planned. |
ycombinator
added a commit
that referenced
this pull request
Mar 26, 2019
Resolves #27513. _This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from #31293: ## Before this PR The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. ## With this PR This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. ## Gains, by the numbers Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
ycombinator
added a commit
to ycombinator/kibana
that referenced
this pull request
Mar 26, 2019
Resolves elastic#27513. _This PR is a combination of elastic#31293 (the code changes) + elastic#33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from elastic#31293: ## Before this PR The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. ## With this PR This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. ## Gains, by the numbers Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
ycombinator
added a commit
to ycombinator/kibana
that referenced
this pull request
Mar 26, 2019
Resolves elastic#27513. _This PR is a combination of elastic#31293 (the code changes) + elastic#33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from elastic#31293: The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
ycombinator
added a commit
to ycombinator/kibana
that referenced
this pull request
Mar 26, 2019
Resolves elastic#27513. _This PR is a combination of elastic#31293 (the code changes) + elastic#33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from elastic#31293: The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
ycombinator
added a commit
that referenced
this pull request
Mar 26, 2019
Resolves #27513. _This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from #31293: ## Before this PR The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. ## With this PR This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. ## Gains, by the numbers Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
ycombinator
added a commit
that referenced
this pull request
Mar 26, 2019
Resolves #27513. _This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from #31293: The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
ycombinator
added a commit
that referenced
this pull request
Mar 26, 2019
Resolves #27513. _This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from #31293: The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
mistic
added a commit
that referenced
this pull request
Mar 26, 2019
* chore(NA): first changes on every package.json order to support new babel 7. chore(NA): build for kbn-pm with babel 7. * chore(NA): patch babel register to load typescrit * chore(NA): first working version with babel 7 replacing typescript compiler. * fix(NA): common preset declaration in order to make it work with babel-loader. * chore(na): organizing babel preset env package json. * chore(NA): mocha tests enabled. * fix(NA): typo on importing * test(NA): majority of x-pack tests ported to use babel-jest * fix(NA): report info button test with babel-jest. * fix(NA): polling service tests. * test(na): fix server plugins plugin tests. * test(NA): batch of test fixs for jest tests under babel-jest hoisting. * chore(NA): add babel plugin to hoist mock prefixed vars on jest tests. * chore(NA): update yarn.lock file. * chore(NA): tests passing. * chore(NA): remove wrong dep * chore(NA): fix tsconfig * chore(NA): skip babel for ts-jest. * chore(NA): selectively apply the plugin to strip off namespace from ts files. * chore(NA): remove not needed changes from ts tests * chore(NA): removed ts-jest dependency. chore(NA): migrate ts tests on x-pack to use babel-jest with the new pattern. * chore(NA): migrate kibana default distribution typescript tests to run with babel-jest and the new test mock pattern. * chore(NA): merge and solve conflicts with master. * chore(NA): fix problems reported by eslint * chore(NA): fix license ovveride for babel-plugin-mock-imports * chore(NA): update jest integration tests for kbn pm * chore(NA): update babel jest integration tests for kbn pm. * test(NA): update jest integration snapshot for kbn pm. * chore(NA): apply changes according to the pull request reviews. * chore(NA): apply changes according to the pull request reviews. * refact(NA): migrate jest tests to the new pattern. * fix(NA): babel 7 polyfill in the tests bundle. * chore(NA): restore needed step in order to compile x-pack with typescript. * chore(NA): change build to compile typescript with babel for the oss code. chore(NA): change transpile typescript task to only transpile types for x-pack. refact(NA): common preset for babel 7 * Revert "chore(NA): change build to compile typescript with babel for the oss code. chore(NA): change transpile typescript task to only transpile types for x-pack. refact(NA): common preset for babel 7" This reverts commit 2707d53. * fix(NA): import paths for tabConfigConst * chore(NA): fix transpiling error on browser tests * chore(NA): simplify kbn babel preset package. * chore(NA): migrate build to use babel transpiler for typescript excluding xpack. * fix(NA): introduced error on test quick task. * fix(NA): fix preset for client side code on build. * fix(NA): build with babel * fix(NA): negated patterns in the end. * fix(NA): kbn_tp_sample_panel_action creation. * fix(NA): babel typescript transform plugin workaround when exporting interface name. * refact(NA): remove not needed type cast to any on jest test. * docs(NA): add developement documentation about jest mocks test pattern. * chore(NA): missing unmerged path. * chore(NA): fix jest tests for template. * [CCR] Client integration tests (table lists) (#33525) * Force user to re-authenticate if token refresh fails with `400` status code. (#33774) * Improve performance of the Logstash Pipeline Viewer (#33793) Resolves #27513. _This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from #31293: ## Before this PR The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. ## With this PR This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. ## Gains, by the numbers Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size. * [Maps] split settings into layer and source panels (#33788) * [Maps] split settings into layer and source panels * fix SCSS import * [env] exit if starting as root (#21563) * [env] exit if starting as root * fix windows * s/--allow-root * Typescript sample panel action (#33602) * Typescript sample panel action * Update EUI version to match main cabana version * update yarn.lock * add back typings include * use correct relative path * Home page "recent links" should communicate saved object type #21896 (#33694) * adds object type for screen order * adds object type for pointer hovering * Update src/legacy/ui/public/chrome/directives/header_global_nav/components/header.tsx Co-Authored-By: rockfield <[email protected]>
joelgriffith
pushed a commit
that referenced
this pull request
Mar 27, 2019
Resolves #27513. _This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from #31293: ## Before this PR The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. ## With this PR This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. ## Gains, by the numbers Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
joelgriffith
pushed a commit
that referenced
this pull request
Mar 27, 2019
* chore(NA): first changes on every package.json order to support new babel 7. chore(NA): build for kbn-pm with babel 7. * chore(NA): patch babel register to load typescrit * chore(NA): first working version with babel 7 replacing typescript compiler. * fix(NA): common preset declaration in order to make it work with babel-loader. * chore(na): organizing babel preset env package json. * chore(NA): mocha tests enabled. * fix(NA): typo on importing * test(NA): majority of x-pack tests ported to use babel-jest * fix(NA): report info button test with babel-jest. * fix(NA): polling service tests. * test(na): fix server plugins plugin tests. * test(NA): batch of test fixs for jest tests under babel-jest hoisting. * chore(NA): add babel plugin to hoist mock prefixed vars on jest tests. * chore(NA): update yarn.lock file. * chore(NA): tests passing. * chore(NA): remove wrong dep * chore(NA): fix tsconfig * chore(NA): skip babel for ts-jest. * chore(NA): selectively apply the plugin to strip off namespace from ts files. * chore(NA): remove not needed changes from ts tests * chore(NA): removed ts-jest dependency. chore(NA): migrate ts tests on x-pack to use babel-jest with the new pattern. * chore(NA): migrate kibana default distribution typescript tests to run with babel-jest and the new test mock pattern. * chore(NA): merge and solve conflicts with master. * chore(NA): fix problems reported by eslint * chore(NA): fix license ovveride for babel-plugin-mock-imports * chore(NA): update jest integration tests for kbn pm * chore(NA): update babel jest integration tests for kbn pm. * test(NA): update jest integration snapshot for kbn pm. * chore(NA): apply changes according to the pull request reviews. * chore(NA): apply changes according to the pull request reviews. * refact(NA): migrate jest tests to the new pattern. * fix(NA): babel 7 polyfill in the tests bundle. * chore(NA): restore needed step in order to compile x-pack with typescript. * chore(NA): change build to compile typescript with babel for the oss code. chore(NA): change transpile typescript task to only transpile types for x-pack. refact(NA): common preset for babel 7 * Revert "chore(NA): change build to compile typescript with babel for the oss code. chore(NA): change transpile typescript task to only transpile types for x-pack. refact(NA): common preset for babel 7" This reverts commit 2707d53. * fix(NA): import paths for tabConfigConst * chore(NA): fix transpiling error on browser tests * chore(NA): simplify kbn babel preset package. * chore(NA): migrate build to use babel transpiler for typescript excluding xpack. * fix(NA): introduced error on test quick task. * fix(NA): fix preset for client side code on build. * fix(NA): build with babel * fix(NA): negated patterns in the end. * fix(NA): kbn_tp_sample_panel_action creation. * fix(NA): babel typescript transform plugin workaround when exporting interface name. * refact(NA): remove not needed type cast to any on jest test. * docs(NA): add developement documentation about jest mocks test pattern. * chore(NA): missing unmerged path. * chore(NA): fix jest tests for template. * [CCR] Client integration tests (table lists) (#33525) * Force user to re-authenticate if token refresh fails with `400` status code. (#33774) * Improve performance of the Logstash Pipeline Viewer (#33793) Resolves #27513. _This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._ Summary of changes, taken from #31293: ## Before this PR The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline. Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser. ## With this PR This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer. ## Gains, by the numbers Based on a simple, 1-input, 1-filter, and 1-output pipeline. * Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets. * Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size. * Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size. * Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size. * [Maps] split settings into layer and source panels (#33788) * [Maps] split settings into layer and source panels * fix SCSS import * [env] exit if starting as root (#21563) * [env] exit if starting as root * fix windows * s/--allow-root * Typescript sample panel action (#33602) * Typescript sample panel action * Update EUI version to match main cabana version * update yarn.lock * add back typings include * use correct relative path * Home page "recent links" should communicate saved object type #21896 (#33694) * adds object type for screen order * adds object type for pointer hovering * Update src/legacy/ui/public/chrome/directives/header_global_nav/components/header.tsx Co-Authored-By: rockfield <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR targets a feature branch instead of
master. Once this PR is merged, a follow up PR will be made to the feature branch to update tests. The idea here is to keep the size of each PR small. Finally, a PR will be created from the feature branch tomaster.Resolves #27513.
Before this PR
The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline.
Naturally, this single API call fetched a lot of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser.
With this PR
This PR makes the Kibana API call to fetch data necessary for initially rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer.
Concretely, this PR makes the following changes:
For each metric of each vertex in the pipeline viewer, instead of returning a timeseries in the Kibana API response, we now only return the latest data point. In other words, instead of returning
[ [t1, v1], [t2, v2], ... [tn, vn] ]we now only returnvn. This dramatically cuts down the number of aggregation buckets that Elasticsearch needs to generate.For each vertex in the pipeline viewer, instead of returning up to 9 metrics in the Kibana API response we now only return 4. These are the 4 that are actually needed for the initial rendering of the pipeline (i.e. with the detail drawer closed). The 5 metrics that are no longer returned are
events_in,events_out,duration_in_millis,queue_push_duration_in_millis, andqueue_push_duration_in_millis_per_event.Since we no longer return
queue_push_duration_in_millis, andqueue_push_duration_in_millis_per_eventfrom the Kibana API response, we also no longer compute the aggregations behind these metrics. Specifically, we no longer need thequeue_push_duration_in_millis_temp_by_ephemeral_idandqueue_push_duration_in_millis_totalaggregations.We no longer return the
pipeline.representation.pluginsarray in the Kibana API response, since this array is not currently used in the Pipeline Viewer UI. We do this by excluding thepluginsfield from the Elasticsearch API response for thelogstash_statedocument.A new API to return timeseries metrics for a single vertex is created. This API is called whenever the detail drawer is open and showing a plugin vertex.
Gains, by the numbers
Based on a simple, 1-input, 1-filter, and 1-output pipeline.
Before this PR, the Elasticsearch
logstash_statsAPI responses (multiple calls were made using thecompositeaggregation over thedate_histogramaggregation) generated a total of 1228 aggregation buckets (before anyfilter_paths were applied but across allcomposite"pages"). With this PR, the singlelogstash_statsAPI response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before anyfilter_paths were applied). That's a 99.02% reduction in number of buckets.Before this PR, the Elasticsearch
logstash_statsAPI responses added up to 70319 bytes. With this PR, the singlelogstash_statsAPI response for the same pipeline is 746 bytes. That's a 98.93% reduction in size.Before this PR, the Elasticsearch
logstash_stateAPI response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a 69.83% reduction in size.Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a 95.04% reduction in size.
Testing this PR
Setup Logstash Monitoring.
Start a Logstash pipeline, like so:
Let the pipeline run for ~30 minutes (due to this bug 😞).
Open the Monitoring UI and visit the Logstash Pipeline Viewer for your pipeline.
Make sure the pipeline shows up. Check that the API response for the initial rendering of the pipeline doesn't contain any timeseries data. Then click on a vertex to open the Detail Drawer. Check that the details for that vertex show up, including sparklines rendered using timeseries data. Sample screenshots of what to expect:
dropfilter plugin vertex