[Security Solution] Update e2e tests to work with detection rules installed from a Fleet package #142311

xcrzx · 2022-09-30T09:30:12Z

Summary

This PR prepares Cypress tests to work with prebuilt detection rules installed from a Fleet package as the filesystem distribution method of prebuilt rules will be removed in future releases (see #139926).

Notable changes:

Before installing prebuilt rules, call waitTillPrepackagedRulesReadyToInstall() to ensure they are available because it could take some time to install the Fleet package with rules (it happens automatically on the initial page load).
Total prebuilt rules count is no longer known. In tests, we should call getPrepackagedRulesCount() to fetch the total number of prebuilt rules via API.

cla-checker-service · 2022-09-30T09:30:16Z

💚 CLA has been signed

elasticmachine · 2022-10-04T14:24:21Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2022-10-04T14:24:23Z

Pinging @elastic/security-solution (Team: SecuritySolution)

banderror · 2022-10-04T17:45:09Z

Files by Code Owner

elastic/security-detections-response

x-pack/plugins/security_solution/cypress/objects/rule.ts

elastic/security-detections-response-alerts

x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/export_rule.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/prebuilt_rules.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/related_integrations.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/sorting.cy.ts

elastic/security-detections-response-rules

x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/export_rule.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/prebuilt_rules.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/related_integrations.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/sorting.cy.ts

elastic/security-solution

x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/export_rule.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/prebuilt_rules.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/related_integrations.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts
x-pack/plugins/security_solution/cypress/e2e/detection_rules/sorting.cy.ts
x-pack/plugins/security_solution/cypress/objects/rule.ts
x-pack/plugins/security_solution/cypress/tasks/alerts_detection_rules.ts
x-pack/plugins/security_solution/cypress/tasks/api_calls/prepackaged_rules.ts
x-pack/plugins/security_solution/cypress/tasks/common.ts

elastic/security-threat-hunting

x-pack/plugins/security_solution/cypress/objects/rule.ts

banderror

Overall LGTM 👍 I left a few relatively minor comments.

I run some of these tests locally and got various timeouts. Now I'm pretty convinced that the default timeout of 20 seconds just doesn't work. We need to increase it to a value that would work in 99% of local test runs for every engineer in AET. I can play with it later and open a separate PR. cc @MadameSheema

banderror · 2022-10-05T17:00:11Z

x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts

+      getPrepackagedRulesCount().then((prepackagedRulesCount) => {
+        checkPrebuiltRulesCannotBeModified(prepackagedRulesCount);
+      });


Nit: Let's be consistent with naming and call it prebuilt everywhere :)

I'd stick to prepackaged, then. As that's how we call them on the API level, and that's something we cannot change already.

As discussed offline, we are keeping the prebuilt naming. However, there still will be some discrepancy between naming in some parts of our application (e.g., schema and API still use prepackaged), which will be addressed separately.

banderror · 2022-10-05T17:10:23Z

x-pack/plugins/security_solution/cypress/e2e/detection_rules/export_rule.cy.ts

+    getPrepackagedRulesCount().then((prepackagedRulesCount) => {
+      const expectedNumberCustomRulesToBeExported = 1;
+      const totalNumberOfRules = expectedNumberCustomRulesToBeExported + prepackagedRulesCount;

-    loadPrebuiltDetectionRulesFromHeaderBtn();
+      loadPrebuiltDetectionRulesFromHeaderBtn();

-    selectAllRules();
-    bulkExportRules();
+      selectAllRules();
+      bulkExportRules();

-    cy.get(MODAL_CONFIRMATION_BODY).contains(
-      `${totalNumberOfPrebuiltRules} prebuilt Elastic rules (exporting prebuilt rules is not supported)`
-    );
+      cy.get(MODAL_CONFIRMATION_BODY).contains(
+        `${prepackagedRulesCount} prebuilt Elastic rules (exporting prebuilt rules is not supported)`
+      );


Here we're getting the count before installing the rules. I guess, it works because getPrepackagedRulesCount returns the number of prebuilt rules (to be precise, the number of security-rule asset SOs) available for installation rather than the number of installed prebuilt rules (the number of alert SOs).

I think this naming can be confusing and I'd rename getPrepackagedRulesCount to getNumberOfAllAvailablePrebuiltRules or smth like that.

Also, shouldn't we wait after calling loadPrebuiltDetectionRulesFromHeaderBtn until the installation is finished? I don't see any explicit check for this like waitForPrebuiltDetectionRulesToBeLoaded

Yes, I see how that naming could be confusing. Changed getPrepackagedRulesCount to getAvailablePrepackagedRulesCount, hope that helps a bit.

Also, shouldn't we wait after calling loadPrebuiltDetectionRulesFromHeaderBtn until the installation is finished?

Inside that method, we wait until the update button disappears, indicating the completion. So we should be good (until UI changes 🙂).

banderror · 2022-10-05T17:21:33Z

x-pack/plugins/security_solution/cypress/e2e/detection_rules/prebuilt_rules.cy.ts

@@ -79,31 +64,27 @@ describe('Prebuilt rules', () => {

        cy.get(SELECT_ALL_RULES_ON_PAGE_CHECKBOX).click();
        enableSelectedRules();
-        waitForRuleToChangeStatus();
+        waitForRuleToUpdate();


Offtopic, but... I'm curious how this test called Allows to enable/disable all rules at once ended up being a part of the Rule monitoring table set which is in turn a part of the test suite for prebuilt rules. How all that is related?

🤷‍♂️ I don't think there's a good reason for that. This code is probably overdue for refactoring.

banderror · 2022-10-05T17:23:30Z

x-pack/plugins/security_solution/cypress/e2e/detection_rules/prebuilt_rules.cy.ts

+  beforeEach(() => {
+    deleteAlertsAndRules();
+    visitWithoutDateRange(DETECTIONS_RULE_MANAGEMENT_URL);
+    loadPrebuiltDetectionRules();
+    waitForPrebuiltDetectionRulesToBeLoaded();
+  });


banderror · 2022-10-05T17:28:28Z

x-pack/plugins/security_solution/cypress/e2e/detection_rules/related_integrations.cy.ts

@@ -184,7 +184,7 @@ describe('Related integrations', () => {
          '{"package":"system","version":"1.17.0"}{"package":"aws","integration":"cloudtrail","version":"1.17.0"}{"package":"aws","integration":"cloudfront","version":"1.17.0"}{"package":"aws","integration":"unknown","version":"1.17.0"}';

        enableRule(firstRule);
-        waitForRuleToChangeStatus();
+        waitForRuleToUpdate();


banderror · 2022-10-05T17:32:06Z

x-pack/plugins/security_solution/cypress/e2e/detection_rules/rules_selection.cy.ts

+    getPrepackagedRulesCount().then((prepackagedRulesCount) => {
+      cy.get(SELECTED_RULES_NUMBER_LABEL).should('contain.text', prepackagedRulesCount);
+    });


Yeah, I can see that loadPrebuiltDetectionRules is used here in the tests for rule selection in the table, and I don't see how that feature is related to prebuilt rules and why we should be loading all of them to test it.

Seems like we'll need to revisit e2e test coverage for the table after we refactor it.

banderror · 2022-10-05T17:40:06Z

x-pack/plugins/security_solution/cypress/tasks/alerts_detection_rules.ts

@@ -132,16 +133,19 @@ export const deleteRuleFromDetailsPage = () => {
 };

 export const duplicateSelectedRules = () => {
+  cy.log('Duplicate selected rules');


Thank you for this improvement 👍

Nit: Can we / should we do it like this to support refactorings easier:

cy.log(duplicateSelectedRules.name)

The downside is probably slightly worse readability of Cypress logs.

We'll also need to rewrite all helpers to named functions for this to work. But I'd prefer log messages to be more easily readable by humans as the test runner's output is already overloaded with technical information.

banderror · 2022-10-05T17:44:07Z

x-pack/plugins/security_solution/cypress/tasks/alerts_detection_rules.ts

+export const waitForRuleToUpdate = () => {
+  cy.log('Wait for rules to update');
+  cy.get(RULE_SWITCH_LOADER, { timeout: 300000 }).should('exist');
+  cy.get(RULE_SWITCH_LOADER, { timeout: 300000 }).should('not.exist');
 };


It would be great to add a comment for this function and clarify what it exactly does and where/when it can be used. Because the name is kinda abstract. Perhaps some renaming wouldn't hurt.

banderror · 2022-10-05T17:48:02Z

x-pack/plugins/security_solution/cypress/tasks/api_calls/prepackaged_rules.ts

+export const getPrepackagedRulesCount = () => {
+  cy.log('Get prepackaged rules count');
+  return getPrepackagedRulesStatus().then(({ body }) => {
+    const prebuiltRulesCount = body.rules_installed + body.rules_not_installed;
+
+    return prebuiltRulesCount;
+  });
+};


For visibility: we talked that this function should return the number of existing security-rule asset saved objects instead of calculating this sum. In order to do this, the api/detection_engine/rules/prepackaged/_status endpoint will need to return this value in its response.

vitaliidm · 2022-10-10T11:31:10Z

x-pack/plugins/security_solution/cypress/e2e/detection_rules/bulk_edit_rules.cy.ts

@@ -210,7 +212,9 @@ describe('Detection rules, bulk edit', () => {
      clickAddTagsMenuItem();
      waitForMixedRulesBulkEditModal(expectedNumberOfCustomRulesToBeEdited);

-      checkPrebuiltRulesCannotBeModified(totalNumberOfPrebuiltRules);
+      getAvailablePrepackagedRulesCount().then((availablePrepackagedRulesCount) => {


inside in getAvailablePrepackagedRulesCoun, there is a checks for:

const prebuiltRulesCount = body.rules_installed + body.rules_not_installed;

but the number of prebuilt rules displayed in UI is the number of installed.
So, correctly would to assert this value against number of installed rules

but the number of prebuilt rules displayed in UI is the number of installed.
So, correctly would to assert this value against number of installed rules

On the loadPrebuiltDetectionRulesFromHeaderBtn() step, all available prebuilt rules are getting installed, so it is okay to check if the UI displays all rules count.

vitaliidm · 2022-10-10T11:36:20Z

x-pack/plugins/security_solution/cypress/tasks/alerts_detection_rules.ts

@@ -179,7 +185,9 @@ export const loadPrebuiltDetectionRules = () => {
 * load prebuilt rules by clicking button on page header
 */
 export const loadPrebuiltDetectionRulesFromHeaderBtn = () => {
-  cy.get(LOAD_PREBUILT_RULES_ON_PAGE_HEADER_BTN).click().should('not.exist');
+  cy.log('load prebuilt detection rules from header');
+  waitTillPrepackagedRulesReadyToInstall();


let's use one name for prepackaged/prebuilt rules.
In some places it's prebuilt, in other prepackaged. Can be confusing, especially for new developers

That's been already discussed: #142311 (comment)

…-ref HEAD~1..HEAD --fix'

kibana-ci · 2022-10-10T13:22:10Z

💚 Build Succeeded

Buildkite Build
Commit: 644fffe

Metrics [docs]

✅ unchanged

History

💛 Build #78715 was flaky 9db70fe
💚 Build #78464 succeeded b7d0cd3
💛 Build #77829 was flaky 4451fa7
💚 Build #77621 succeeded dd3ee11
💔 Build #77581 failed 7081396
💔 Build #77466 failed 73754fc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

xcrzx added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Sep 30, 2022

xcrzx self-assigned this Sep 30, 2022

xcrzx force-pushed the test-prebuilt-rules-from-registry branch 8 times, most recently from 7081396 to 35a1707 Compare October 3, 2022 16:24

xcrzx changed the title ~~[Security Solution] Tesk Kibana with prebuilt rules from a remote repository~~ [Security Solution] Test Kibana with prebuilt rules from a remote repository Oct 4, 2022

xcrzx force-pushed the test-prebuilt-rules-from-registry branch from dd3ee11 to 2e5019c Compare October 4, 2022 09:08

xcrzx marked this pull request as ready for review October 4, 2022 14:24

xcrzx requested review from a team as code owners October 4, 2022 14:24

xcrzx requested a review from banderror October 4, 2022 14:24

xcrzx added release_note:skip Skip the PR/issue when compiling release notes v8.6.0 Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules labels Oct 4, 2022

xcrzx force-pushed the test-prebuilt-rules-from-registry branch from 4451fa7 to a3e15ce Compare October 5, 2022 16:25

banderror reviewed Oct 5, 2022

View reviewed changes

xcrzx force-pushed the test-prebuilt-rules-from-registry branch from b7d0cd3 to 1c52554 Compare October 6, 2022 13:36

banderror approved these changes Oct 10, 2022

View reviewed changes

xcrzx changed the title ~~[Security Solution] Test Kibana with prebuilt rules from a remote repository~~ [Security Solution] Update e2e tests to work with detection rules installed from a Fleet package Oct 10, 2022

vitaliidm reviewed Oct 10, 2022

View reviewed changes

Tesk Kibana with prebuilt rules from a remote repository

c40a789

xcrzx force-pushed the test-prebuilt-rules-from-registry branch from 9db70fe to c40a789 Compare October 10, 2022 11:58

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

644fffe

…-ref HEAD~1..HEAD --fix'

xcrzx merged commit 3d14629 into elastic:main Oct 10, 2022

kibanamachine added the backport:skip This commit does not require backporting label Oct 10, 2022

xcrzx deleted the test-prebuilt-rules-from-registry branch October 10, 2022 13:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Update e2e tests to work with detection rules installed from a Fleet package #142311

[Security Solution] Update e2e tests to work with detection rules installed from a Fleet package #142311

xcrzx commented Sep 30, 2022 •

edited

Loading

cla-checker-service bot commented Sep 30, 2022 •

edited

Loading

elasticmachine commented Oct 4, 2022

elasticmachine commented Oct 4, 2022

banderror commented Oct 4, 2022

banderror left a comment •

edited

Loading

banderror Oct 5, 2022

xcrzx Oct 6, 2022

xcrzx Oct 10, 2022

banderror Oct 5, 2022

xcrzx Oct 6, 2022

banderror Oct 5, 2022

xcrzx Oct 6, 2022

banderror Oct 5, 2022

banderror Oct 5, 2022

banderror Oct 5, 2022

banderror Oct 5, 2022

xcrzx Oct 6, 2022

banderror Oct 5, 2022

banderror Oct 5, 2022

vitaliidm Oct 10, 2022 •

edited

Loading

xcrzx Oct 10, 2022

vitaliidm Oct 10, 2022

xcrzx Oct 10, 2022

kibana-ci commented Oct 10, 2022

[Security Solution] Update e2e tests to work with detection rules installed from a Fleet package #142311

[Security Solution] Update e2e tests to work with detection rules installed from a Fleet package #142311

Conversation

xcrzx commented Sep 30, 2022 • edited Loading

Summary

cla-checker-service bot commented Sep 30, 2022 • edited Loading

elasticmachine commented Oct 4, 2022

elasticmachine commented Oct 4, 2022

banderror commented Oct 4, 2022

Files by Code Owner

elastic/security-detections-response

elastic/security-detections-response-alerts

elastic/security-detections-response-rules

elastic/security-solution

elastic/security-threat-hunting

banderror left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitaliidm Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Oct 10, 2022

💚 Build Succeeded

Metrics [docs]

History

xcrzx commented Sep 30, 2022 •

edited

Loading

cla-checker-service bot commented Sep 30, 2022 •

edited

Loading

banderror left a comment •

edited

Loading

vitaliidm Oct 10, 2022 •

edited

Loading