[Fleet] refactor auto upgrade package policies logic #125909

juliaElastic · 2022-02-17T08:49:55Z

Summary

Refactored logic of auto upgrade package policies:

querying installed packages with keep_policies_up_to_date flag, then
query package policies where version != package version
filter out package policies where version >= package version
run dry run and then run upgrade if no errors

Separated auto upgrade package policy logic from ensure preconfigured policies logic, so that it is no longer called when invoking API POST /api/fleet/setup/preconfiguration

To verify (existing functionality):

install old version of system package

POST http://elastic:changeme@localhost:5620/api/fleet/epm/packages/system-1.9.0
kbn-xsrf: kibana

{ "force": true }

add system integration
switch keep policies up to date on (integration settings page)
update package to latest without policies
navigate to Fleet app (or POST /api/fleet/setup)
verify that system package policy is upgraded

Checklist

Unit or functional tests were updated or added to match the most common scenarios

x-pack/plugins/fleet/server/services/setup.ts

x-pack/plugins/fleet/server/services/managed_package_policies.ts

joshdover · 2022-02-17T13:58:24Z

x-pack/plugins/fleet/server/services/managed_package_policies.ts

+  // Since upgrades don't report diffs/errors, we need to perform a dry run first in order
+  // to notify the user of any granular policy upgrade errors that occur during Fleet's
+  // preconfiguration check
+  const dryRunResults = await packagePolicyService.getUpgradeDryRunDiff(soClient, packagePolicyId);


This getUpgradeDryRunDiff function may have some opportunities for other improvements. It currently makes these queries:

Within getUpgradePackagePolicyInfo

It fetches the package policy again (we already have it fetched above)

It fetches getInstallation again (we already have it fetched above)

It fetches getPackageInfo - this has many of the same fields that Installation does

It fetches the package from the registry Registry.fetchInfo - the data needed here could be compiled from the package info I believe. Related to [Fleet] Rely on the content of a package when installing it #115032

Worth noting that this upgrade function is not the main pain-point for users since this should only happen once on startup and not really need to be processed again. For users with 1000s of package policies, the improvements in this PR and #125788 should already be a big gain.

Good points, for this pr I will keep the original scope and related tests.

I can raise another pr to take a stab at optimizing getUpgradeDryRunDiff. One thing I noticed that it is used from the handler of dry run API too, where those variables are not loaded. So it has to be separated to 2 functions to optimize for setup use case.

So it has to be separated to 2 functions to optimize for setup use case.

Yeah it'd probably make sense to have a function that only does the processing given the already loaded objects and then a function that does the loading and calls the first function.

elasticmachine · 2022-02-17T13:58:49Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2022-02-21T07:34:10Z

@elasticmachine merge upstream

kibana-ci · 2022-02-21T11:08:33Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #25206 succeeded 25b22a5
💚 Build #25021 succeeded eae6105
💚 Build #24954 succeeded 7b75a22
💚 Build #24923 succeeded f40c881
💔 Build #24899 failed cae61df

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

* refactor upgrade package policies * fixed tests * code cleanup * review improvements * added api test Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2022-02-23T11:36:18Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 125909 or prevent reminders by adding the backport:skip label.

* refactor upgrade package policies * fixed tests * code cleanup * review improvements * added api test Co-authored-by: Kibana Machine <[email protected]>

* WIP * WIP2 * Use new cases context hooks to open and close the flyout * Update timelines to use new hooks * CLose flyout on create success * Add back sucess toast * Move code to a dedicated component * Add CasesContext to observability * Remove dependency * Small refactor * Use observabilityAppId instead of observabilityFeatureId for buttons * Add CasesContext to timetable * Fix detection engine test cases * Fix broken tests * Fix broken tests * Rename hook * Add test cases for cases context ui * Add test for new hook * Remove state from the provider context * Remove basevalue * apply suggested renaming * Add usecallback * Add reducer types, fix test type, remove redundant check * Accept attachments as a prop for the cases select modal * Expose useCasesAddToExistingCase hook, reducer code and global component * use the new hook to open the select cases modasl * Fix tests and types * Add tests for cases global components * [Fleet] showing agent policy creation error message on UI (#125931) * showing agent policy creation error message on UI * mapping the error instead of showing from the backend Co-authored-by: Kibana Machine <[email protected]> * [ResponseOps] Adds tooltip to time window selector in ES query rule flyout (#125764) * [Lens] Allow detaching from global time range (#125563) * allow detaching from global time range * add test * fix time field recognition * fix tests Co-authored-by: Kibana Machine <[email protected]> * [Fleet] refactor auto upgrade package policies logic (#125909) * refactor upgrade package policies * fixed tests * code cleanup * review improvements * added api test Co-authored-by: Kibana Machine <[email protected]> * skip flaky suite (#126027) * Remove deprecated api (#125524) * [Fleet] Remove deprecated kibana APIs - License * Remove basePath from FleetApp * Replace AsyncPlugin with Plugin * Get fieldFormats from fieldFormats plugin rather than data plugin * Fix ts errors * Attempt fixing wrong type * Move licenseService to FleetStartDeps * Fix types and mocks Co-authored-by: Kibana Machine <[email protected]> * Upgrade `markdown-it` dependency (`10.0.0` → `12.3.2`). (#125526) * skipping failing tests (#126039) * remove unused deprecated code and use field format plugin directly for data view field editor (#126029) * [data views] Improve preview pane (#126013) * fix preview pane * fix preview pane * one less span tag Co-authored-by: Kibana Machine <[email protected]> * [Alerting] Provide services to set context for recovered alerts (#124972) * Rename alert instance to alert and add create fn to alert factory * Rename alert instance to alert and add create fn to alert factory * Fixing types * Fixing types * Adding flag for rule types to opt into setting recovery context * Only showing context in action variable menu if flag set to true * Adding recovery functions to createAlertFactory * Setting recovery in index threshold and fixing types * Fixing lint issues and some refactoring * Cleanup * Functional tests for index threshold rule recovery context * Return array of recovered alerts instead of record * Fixing types * Fixing types * Cleanup * Handling nulls and more tests * Updating developer docs * Making getRecoveryAlerts non-optional * Setting unknown in index threshold recovery value * PR feedback * Adding a test Co-authored-by: Kibana Machine <[email protected]> * [Discover] Re-introduce saved_searches test (#126059) * [Archive Migration] index pattern without timefield (#125870) * kbn_archive date_nanos * kbn_archive date_nanos in context and discover * kbn_archiver more date_nanos tests * split out kbnArchiver for index_pattern_without_timefield * remove date_nanos files from a different PR * update another test for usage of the same archives * set default index pattern for test * remove duplicate const kibanaServer Co-authored-by: Kibana Machine <[email protected]> * delete unused es_archive visualize_embedding (#126001) * delete unused es_archive * remove other unused es_archive * more unused es_archives Co-authored-by: Kibana Machine <[email protected]> * Bump packages (#126119) * url-parse 1.5.3 -> 1.5.9 * follow-redirects 1.y.z -> 1.14.9 * Add tests for all cases selector and attachments * Add tests for use add to existing case hook * First version of the cases timeline actions * export add alert to new case button from cases plugin * Make Cases ECS compatible with timelines and security_solution * Delete new case button * Add helpers * Use the cases hook directly for add to new case * Remov unused dependencies * Rename callbacks, remove timelines calls * Fixing tests for the dropdown * Fix broken test * mocking cases for tests * Fix detectiosn tests * Observability now uses the new cases hooks * Wrap events viewer into cases context * Open the create case flyout if create case was selected in the modal * Fix cases mocks for security_solution * Update tests * Add tests for use cases toast * Improve cases mock * delete security mock * replace tests mocks for cases * fix import mock * Do not require onRowClick * Show the toast inside the modal * show the toast inside the flyout * remove toast logic from the consumer plugin * fix typescript types * Rename type * Fix broken test * Fix file name and broken test * Use internal navigation hook * Update hook dependencies * Move useCaseToast * Fix mock paths * fix eslint * Add test cases for the toast content * Add cases context to the overview page Co-authored-by: Julia Bardi <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: István Zoltán Szabó <[email protected]> Co-authored-by: Joe Reuter <[email protected]> Co-authored-by: Tiago Costa <[email protected]> Co-authored-by: Cristina Amico <[email protected]> Co-authored-by: Aleh Zasypkin <[email protected]> Co-authored-by: Gloria Hornero <[email protected]> Co-authored-by: Matthew Kime <[email protected]> Co-authored-by: Ying Mao <[email protected]> Co-authored-by: Maja Grubic <[email protected]> Co-authored-by: Lee Drengenberg <[email protected]> Co-authored-by: Joe Portner <[email protected]>

…#126265) * WIP * WIP2 * Use new cases context hooks to open and close the flyout * Update timelines to use new hooks * CLose flyout on create success * Add back sucess toast * Move code to a dedicated component * Add CasesContext to observability * Remove dependency * Small refactor * Use observabilityAppId instead of observabilityFeatureId for buttons * Add CasesContext to timetable * Fix detection engine test cases * Fix broken tests * Fix broken tests * Rename hook * Add test cases for cases context ui * Add test for new hook * Remove state from the provider context * Remove basevalue * apply suggested renaming * Add usecallback * Add reducer types, fix test type, remove redundant check * Accept attachments as a prop for the cases select modal * Expose useCasesAddToExistingCase hook, reducer code and global component * use the new hook to open the select cases modasl * Fix tests and types * Add tests for cases global components * [Fleet] showing agent policy creation error message on UI (elastic#125931) * showing agent policy creation error message on UI * mapping the error instead of showing from the backend Co-authored-by: Kibana Machine <[email protected]> * [ResponseOps] Adds tooltip to time window selector in ES query rule flyout (elastic#125764) * [Lens] Allow detaching from global time range (elastic#125563) * allow detaching from global time range * add test * fix time field recognition * fix tests Co-authored-by: Kibana Machine <[email protected]> * [Fleet] refactor auto upgrade package policies logic (elastic#125909) * refactor upgrade package policies * fixed tests * code cleanup * review improvements * added api test Co-authored-by: Kibana Machine <[email protected]> * skip flaky suite (elastic#126027) * Remove deprecated api (elastic#125524) * [Fleet] Remove deprecated kibana APIs - License * Remove basePath from FleetApp * Replace AsyncPlugin with Plugin * Get fieldFormats from fieldFormats plugin rather than data plugin * Fix ts errors * Attempt fixing wrong type * Move licenseService to FleetStartDeps * Fix types and mocks Co-authored-by: Kibana Machine <[email protected]> * Upgrade `markdown-it` dependency (`10.0.0` → `12.3.2`). (elastic#125526) * skipping failing tests (elastic#126039) * remove unused deprecated code and use field format plugin directly for data view field editor (elastic#126029) * [data views] Improve preview pane (elastic#126013) * fix preview pane * fix preview pane * one less span tag Co-authored-by: Kibana Machine <[email protected]> * [Alerting] Provide services to set context for recovered alerts (elastic#124972) * Rename alert instance to alert and add create fn to alert factory * Rename alert instance to alert and add create fn to alert factory * Fixing types * Fixing types * Adding flag for rule types to opt into setting recovery context * Only showing context in action variable menu if flag set to true * Adding recovery functions to createAlertFactory * Setting recovery in index threshold and fixing types * Fixing lint issues and some refactoring * Cleanup * Functional tests for index threshold rule recovery context * Return array of recovered alerts instead of record * Fixing types * Fixing types * Cleanup * Handling nulls and more tests * Updating developer docs * Making getRecoveryAlerts non-optional * Setting unknown in index threshold recovery value * PR feedback * Adding a test Co-authored-by: Kibana Machine <[email protected]> * [Discover] Re-introduce saved_searches test (elastic#126059) * [Archive Migration] index pattern without timefield (elastic#125870) * kbn_archive date_nanos * kbn_archive date_nanos in context and discover * kbn_archiver more date_nanos tests * split out kbnArchiver for index_pattern_without_timefield * remove date_nanos files from a different PR * update another test for usage of the same archives * set default index pattern for test * remove duplicate const kibanaServer Co-authored-by: Kibana Machine <[email protected]> * delete unused es_archive visualize_embedding (elastic#126001) * delete unused es_archive * remove other unused es_archive * more unused es_archives Co-authored-by: Kibana Machine <[email protected]> * Bump packages (elastic#126119) * url-parse 1.5.3 -> 1.5.9 * follow-redirects 1.y.z -> 1.14.9 * Add tests for all cases selector and attachments * Add tests for use add to existing case hook * First version of the cases timeline actions * export add alert to new case button from cases plugin * Make Cases ECS compatible with timelines and security_solution * Delete new case button * Add helpers * Use the cases hook directly for add to new case * Remov unused dependencies * Rename callbacks, remove timelines calls * Fixing tests for the dropdown * Fix broken test * mocking cases for tests * Fix detectiosn tests * Observability now uses the new cases hooks * Wrap events viewer into cases context * Open the create case flyout if create case was selected in the modal * Fix cases mocks for security_solution * Update tests * Add tests for use cases toast * Improve cases mock * delete security mock * replace tests mocks for cases * fix import mock * Do not require onRowClick * Show the toast inside the modal * show the toast inside the flyout * remove toast logic from the consumer plugin * fix typescript types * Rename type * Fix broken test * Fix file name and broken test * Use internal navigation hook * Update hook dependencies * Move useCaseToast * Fix mock paths * fix eslint * Add test cases for the toast content * Add cases context to the overview page Co-authored-by: Julia Bardi <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: István Zoltán Szabó <[email protected]> Co-authored-by: Joe Reuter <[email protected]> Co-authored-by: Tiago Costa <[email protected]> Co-authored-by: Cristina Amico <[email protected]> Co-authored-by: Aleh Zasypkin <[email protected]> Co-authored-by: Gloria Hornero <[email protected]> Co-authored-by: Matthew Kime <[email protected]> Co-authored-by: Ying Mao <[email protected]> Co-authored-by: Maja Grubic <[email protected]> Co-authored-by: Lee Drengenberg <[email protected]> Co-authored-by: Joe Portner <[email protected]>

refactor upgrade package policies

cae61df

juliaElastic self-assigned this Feb 17, 2022

juliaElastic added release_note:enhancement v8.2.0 labels Feb 17, 2022

juliaElastic added 2 commits February 17, 2022 11:45

fixed tests

f40c881

code cleanup

7b75a22

juliaElastic marked this pull request as ready for review February 17, 2022 13:08

juliaElastic requested a review from a team as a code owner February 17, 2022 13:08

joshdover reviewed Feb 17, 2022

View reviewed changes

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Feb 17, 2022

review improvements

eae6105

kibanamachine and others added 2 commits February 21, 2022 02:34

Merge branch 'main' into perf-package-policy-auto-upgrade

25b22a5

added api test

fc2e401

juliaElastic requested a review from joshdover February 21, 2022 10:47

joshdover approved these changes Feb 21, 2022

View reviewed changes

juliaElastic merged commit 4efc9c0 into elastic:main Feb 21, 2022

juliaElastic deleted the perf-package-policy-auto-upgrade branch February 21, 2022 11:36

juliaElastic mentioned this pull request Feb 21, 2022

[Fleet] optimizing package policy upgrade and dry run #126088

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 23, 2022

juliaElastic added backport:skip This commit does not require backporting and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Feb 23, 2022

jen-huang added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:enhancement labels Apr 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] refactor auto upgrade package policies logic #125909

[Fleet] refactor auto upgrade package policies logic #125909

juliaElastic commented Feb 17, 2022 •

edited

Loading

joshdover Feb 17, 2022 •

edited

Loading

joshdover Feb 17, 2022

juliaElastic Feb 17, 2022

joshdover Feb 21, 2022

elasticmachine commented Feb 17, 2022

juliaElastic commented Feb 21, 2022

kibana-ci commented Feb 21, 2022

kibanamachine commented Feb 23, 2022

[Fleet] refactor auto upgrade package policies logic #125909

[Fleet] refactor auto upgrade package policies logic #125909

Conversation

juliaElastic commented Feb 17, 2022 • edited Loading

Summary

Checklist

joshdover Feb 17, 2022 • edited Loading

Choose a reason for hiding this comment

joshdover Feb 17, 2022

Choose a reason for hiding this comment

juliaElastic Feb 17, 2022

Choose a reason for hiding this comment

joshdover Feb 21, 2022

Choose a reason for hiding this comment

elasticmachine commented Feb 17, 2022

juliaElastic commented Feb 21, 2022

kibana-ci commented Feb 21, 2022

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Feb 23, 2022

juliaElastic commented Feb 17, 2022 •

edited

Loading

joshdover Feb 17, 2022 •

edited

Loading