[FTR] support "deployment agnostic" api-integration tests #188737

dmlemeshko · 2024-07-19T11:14:04Z

Summary

Closed in favour of #189853

This PR adds capability to create a single test file and run it against both serverless project and stateful deployment.

How it works:

svlUserManager service is using new SamlAuthProvider service for SAML authentication for serverless and stateful both locally and in Elastic Cloud
DeploymentAgnosticFtrProviderContext limits the scope of FTR services only the ones that compatible in both serverless & stateful
Create a single file and two FTR config files that extends defaults configs for serverless and stateful with SAML support

test example

export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
  const samlAuth = getService('samlAuth');
  const supertestWithoutAuth = getService('supertestWithoutAuth');
  let roleAuthc: RoleCredentials;

  describe('Painless Lab Routes', function () {
    before(async () => {
      roleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('admin');
    });

    describe('Execute', () => {
      it('should execute a valid painless script', async () => {
        const script =
          '"{\\n  \\"script\\": {\\n    \\"source\\": \\"return true;\\",\\n    \\"params\\": {\\n  \\"string_parameter\\": \\"string value\\",\\n  \\"number_parameter\\": 1.5,\\n  \\"boolean_parameter\\": true\\n}\\n  }\\n}"';

        const { body } = await supertestWithoutAuth
          .post(`${API_BASE_PATH}/execute`)
          .set(samlAuth.getInternalRequestHeader())
          .set(roleAuthc.apiKeyHeader)
          .set('Content-Type', 'application/json;charset=UTF-8')
          .send(script)
          .expect(200);

        expect(body).to.eql({
          result: 'true',
        });
      });
      ...
    });
  });
}

stateful config example:

export default createStatefulTestConfig({
  testFiles: [require.resolve('.')],
  junit: {
    reportName: 'Stateful - Painless lab - API Integration Tests',
  },
  // extra arguments
  esServerArgs: [],
  kbnServerArgs: [],
});

serverless Oblt example

export default createServerlessTestConfig({
  serverlessProject: 'oblt',
  testFiles: [require.resolve('.')],
  junit: {
    reportName: 'Serverless Observability - Painless lab - API Integration Tests',
  },

  // include settings from project controller
  // https://github.com/elastic/project-controller/blob/main/internal/project/observability/config/elasticsearch.yml
  esServerArgs: ['xpack.ml.dfa.enabled=false'],
  kbnServerArgs: [
    // defined in MKI control plane
    '--xpack.uptime.service.manifestUrl=mockDevUrl',
    // useful for testing (also enabled in MKI QA)
    '--coreApp.allowDynamicConfigOverrides=true',
  ],
});

Code changes:
This PR modifies serveless-specific SvlUserManagerProvider into deployment agnostic SamlAuthProvider and moves service into kbn-ftr-common-functional-services package. Though it is a "common" service, it shouldn't be loaded in tests that use basic authentication (more details in comment below)

In oder to guarantee the test to work in both serverless and stateful environments (especially MKI and cloud deployments), it is important for every used FTR service to be "deployment-agnostic". SamlAuthProvider can be used as example how to create one. Since FTR config schema has serverless property, service can have 2 implementations of the same interface. This way test code remains "clean" keeping all the difference inside the service.

As a required dependency, SupertestWithoutAuthProvider for SvlUserManagerProvider service was move to the same package in #189613

To simplify new FTR configs creation for serverless and stateful context 2 functions were added:

createServerlessTestConfig(..)
createStatefulTestConfig(..)

…-service

dmlemeshko · 2024-07-22T09:09:24Z

x-pack/plugins/security_solution/public/management/cypress/support/saml_authentication.ts

+        hostOptions,
+        log,
+        isCloud: config.env.CLOUD_SERVERLESS,
+        cloudUsersFilePath: `${REPO_ROOT}/.ftr/${rolesFilename}`,


For some reason not able to use path.resolve here: Do not import Node.js builtin module "path"

Let me know if there is the way to do it in a reliable way (this code will fail on Windows)

Regarding change: I believe passing the absolute path to the file with credentials gives a better flexibility and also less logic inside SamlSessionManager, which now will be used for both svl & stateful

dmlemeshko · 2024-07-22T10:08:27Z

/ci

…fix'

dmlemeshko · 2024-07-22T15:33:26Z

/ci

…-service

dmlemeshko · 2024-07-25T14:57:19Z

/ci

dmlemeshko · 2024-07-26T09:16:17Z

/ci

dmlemeshko · 2024-07-26T12:39:58Z

/ci

…-service

dmlemeshko · 2024-07-26T17:02:25Z

/ci

…-service

dmlemeshko · 2024-07-28T10:35:48Z

/ci

This reverts commit 22fedf1.

…ibana into ftr/unify-saml-auth-service

dmlemeshko · 2024-08-02T11:04:03Z

/ci

…fix'

dmlemeshko · 2024-08-02T12:37:45Z

/ci

mgiota · 2024-08-02T14:10:15Z

packages/kbn-ftr-common-functional-services/services/saml_auth/get_auth_provider.ts

+import { KibanaServer } from '../..';
+
+import { ServerlessAuthProvider } from './serverless/auth_provider';
+import { StatefuAuthProvider } from './stateful/auth_provider';


There is a typo here. Should be StatefulAuthProvider

dmlemeshko · 2024-08-02T15:12:02Z

/ci

…ibana into ftr/unify-saml-auth-service

dmlemeshko · 2024-08-02T15:47:40Z

/ci

dmlemeshko · 2024-08-02T18:12:29Z

/ci

…fix'

…ibana into ftr/unify-saml-auth-service

dmlemeshko · 2024-08-02T22:15:32Z

/ci

kibana-ci · 2024-08-02T23:00:00Z

⏳ Build in-progress, with failures

Failed CI Steps

Test Failures

[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #1 / Add exception using data views from rule details "before each" hook for "Creates an exception item and close all matching alerts" "before each" hook for "Creates an exception item and close all matching alerts"
[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #2 / Add multiple conditions and validate the generated exceptions "before each" hook for "Use multipe AND conditions and validate it generates one exception" "before each" hook for "Use multipe AND conditions and validate it generates one exception"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #1 / Alert details expandable flyout left panel entities "before each" hook for "should display host details and user details under Insights Entities" "before each" hook for "should display host details and user details under Insights Entities"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Alert details expandable flyout left panel investigation "before each" hook for "should display empty response message" "before each" hook for "should display empty response message"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Alert details expandable flyout left panel investigation "before each" hook for "should display investigation guide" "before each" hook for "should display investigation guide"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Alert details expandable flyout left panel prevalence "before each" hook for "should display prevalence tab" "before each" hook for "should display prevalence tab"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #1 / Alert details expandable flyout right panel "before each" hook for "should display header and footer basics" "before each" hook for "should display header and footer basics"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Alert details expandable flyout right panel json tab "before each" hook for "should display the json component" "before each" hook for "should display the json component"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Alert details expandable flyout right panel overview tab about section "before each" hook for "should display about section" "before each" hook for "should display about section"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Alert details expandable flyout right panel table tab "before each" hook for "should display and filter the table" "before each" hook for "should display and filter the table"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Alert details expandable flyout rule preview panel alert reason preview "before each" hook for "should display alert reason preview" "before each" hook for "should display alert reason preview"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / Alert details expandable flyout rule preview panel rule preview "before each" hook for "should display rule preview and its sub sections" "before each" hook for "should display rule preview and its sub sections"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Alert Table API calls "before each" hook for "should call api/lists/index only once" "before each" hook for "should call api/lists/index only once"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #4 / Alert tagging "before each" hook for "Add and remove a tag using the alert bulk action menu" "before each" hook for "Add and remove a tag using the alert bulk action menu"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #1 / Alerts cell actions "before each" hook for "should filter for and out" "before each" hook for "should filter for and out"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Alerts generated by building block rules "before each" hook for "Alerts should be visible on the Rule Detail page and not visible on the Overview page" "before each" hook for "Alerts should be visible on the Rule Detail page and not visible on the Overview page"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #1 / All hosts table with new risk score "before each" hook for "it renders risk column" "before each" hook for "it renders risk column"
[job] [logs] Serverless AI Assistant - Security Solution Cypress Tests / App Features for Security Complete "before each" hook for "should have have AI Assistant available" "before each" hook for "should have have AI Assistant available"
[job] [logs] Serverless AI Assistant - Security Solution Cypress Tests / App Features for Security Essentials "before each" hook for "should not have AI Assistant available" "before each" hook for "should not have AI Assistant available"
[job] [logs] FTR Configs #89 / Application Usage keys in the schema match the registered application IDs
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / attach timeline to case with cases created "before each" hook for "attach timeline to an existing case" "before each" hook for "attach timeline to an existing case"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / attach timeline to case without cases created "before each" hook for "attach timeline to a new case" "before each" hook for "attach timeline to a new case"
[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #2 / Auto populate exception with Alert data "before each" hook for "Should create a Rule exception item from alert actions overflow menu and auto populate the conditions using alert Highlighted fields" "before each" hook for "Should create a Rule exception item from alert actions overflow menu and auto populate the conditions using alert Highlighted fields"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #2 / Backfill groups "before all" hook for "should show backfill groups" "before all" hook for "should show backfill groups"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #4 / Bulk editing index patterns of rules with a data view only "before each" hook for "Add index patterns to custom rules with configured data view: all rules are skipped" "before each" hook for "Add index patterns to custom rules with configured data view: all rules are skipped"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #4 / Bulk editing index patterns of rules with index patterns and rules with a data view "before each" hook for "Add index patterns to custom rules: one rule is updated, one rule is skipped" "before each" hook for "Add index patterns to custom rules: one rule is updated, one rule is skipped"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Bulk Investigate in Timeline Alerts "before each" hook for "Adding multiple alerts to the timeline should be successful" "before each" hook for "Adding multiple alerts to the timeline should be successful"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Bulk Investigate in Timeline Host -> Events Viewer "before each" hook for "Adding multiple events to the timeline should be successful" "before each" hook for "Adding multiple events to the timeline should be successful"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Bulk Investigate in Timeline Host -> Sessions Viewer "before each" hook for "Adding multiple events to the timeline should be successful" "before each" hook for "Adding multiple events to the timeline should be successful"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #2 / Cases connector incident fields "before each" hook for "Correct incident fields show when connector is changed" "before each" hook for "Correct incident fields show when connector is changed"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Cases Creates a new case with timeline and opens the timeline Creates a new case with timeline and opens the timeline
[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #1 / Close matching Alerts "before each" hook for "Should create a Rule exception item from alert actions overflow menu and close all matching alerts" "before each" hook for "Should create a Rule exception item from alert actions overflow menu and close all matching alerts"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #3 / Common rule creation flows "before each" hook for "Creates and enables a rule" "before each" hook for "Creates and enables a rule"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #3 / Common rule detail flows "before each" hook for "Only modifies rule active status on enable/disable" "before each" hook for "Only modifies rule active status on enable/disable"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Correlation tab "before each" hook for "should update timeline after adding eql" "before each" hook for "should update timeline after adding eql"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #5 / Create custom query rule Custom detection rules creation "before each" hook for "Creates and enables a rule" "before each" hook for "Creates and enables a rule"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #5 / Custom query rules Basics "before each" hook for "Allows a rule to be edited" "before each" hook for "Allows a rule to be edited"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Detection ES|QL rules, creation Alert suppression "before each" hook for "shows custom ES|QL field in investigation fields autocomplete and saves it in rule" "before each" hook for "shows custom ES|QL field in investigation fields autocomplete and saves it in rule"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Detection ES|QL rules, creation creation "before each" hook for "creates an ES|QL rule" "before each" hook for "creates an ES|QL rule"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Detection ES|QL rules, creation ES|QL investigation fields "before each" hook for "shows custom ES|QL field in investigation fields autocomplete and saves it in rule" "before each" hook for "shows custom ES|QL field in investigation fields autocomplete and saves it in rule"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Detection ES|QL rules, creation ES|QL query validation "before each" hook for "shows error when ES|QL query is empty" "before each" hook for "shows error when ES|QL query is empty"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #4 / Detection ES|QL rules, details view "before each" hook for "displays ES|QL rule specific fields" "before each" hook for "displays ES|QL rule specific fields"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #4 / Detection Rule Creation - EQL Rules - With Alert Suppression with non-sequence queries "before each" hook for "creates a rule with a "per rule execution" suppression duration" "before each" hook for "creates a rule with a "per rule execution" suppression duration"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #5 / Detection Rule Creation - EQL Rules - With Alert Suppression with sequence queries "before each" hook for "disables the suppression fields and presents an informative tooltip" "before each" hook for "disables the suppression fields and presents an informative tooltip"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #4 / Detection rules, Alert Suppression for Essentials tier "before each" hook for "Alert suppression is enabled for essentials tier for rule types that support it" "before each" hook for "Alert suppression is enabled for essentials tier for rule types that support it"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #1 / Detection rules, bulk duplicate "before each" hook for "Duplicates rules" "before each" hook for "Duplicates rules"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #3 / Detection rules, bulk edit of rule actions Restricted action privileges "before each" hook for "User with no privileges can't add rule actions" "before each" hook for "User with no privileges can't add rule actions"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #2 / Detection rules, bulk edit Prerequisites "before each" hook for "No rules selected" "before each" hook for "No rules selected"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #5 / Detection rules, Indicator Match, Alert Suppression "before each" hook for "creates rule with per rule execution suppression for essentials license" "before each" hook for "creates rule with per rule execution suppression for essentials license"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #4 / Detection rules, Indicator Match, Alert Suppression "before each" hook for "creates rule with per rule execution suppression" "before each" hook for "creates rule with per rule execution suppression"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Detection rules, Indicator Match, Edit without suppression "before each" hook for "enables suppression on time interval" "before each" hook for "enables suppression on time interval"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #4 / Detection rules, New terms, Edit with suppression configured "before each" hook for "displays suppress options correctly on edit form and allows its editing" "before each" hook for "displays suppress options correctly on edit form and allows its editing"
[job] [logs] Serverless Rule Management - Prebuilt Rules - Security Solution Cypress Tests / Detection rules, Prebuilt Rules Installation and Update - Error handling Installation of prebuilt rules - Should fail gracefully with toast error message when "before each" hook for "installing prebuilt rules one by one" "before each" hook for "installing prebuilt rules one by one"
[job] [logs] Serverless Rule Management - Prebuilt Rules - Security Solution Cypress Tests / Detection rules, Prebuilt Rules Installation and Update Notifications No notifications "before each" hook for "should NOT display install or update notifications when no prebuilt assets and no rules are installed" "before each" hook for "should NOT display install or update notifications when no prebuilt assets and no rules are installed"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #5 / Detection threshold rules, edit with suppression enabled "before each" hook for "displays suppress options correctly on edit form" "before each" hook for "displays suppress options correctly on edit form"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #5 / Detection threshold rules, edit without suppression "before each" hook for "enables suppression on time interval" "before each" hook for "enables suppression on time interval"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #2 / Display not found page "before each" hook for "navigates to the alerts page with incorrect link" "before each" hook for "navigates to the alerts page with incorrect link"
[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #2 / Duplicate List "before each" hook for "Duplicate exception list with expired items" "before each" hook for "Duplicate exception list with expired items"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #3 / Enable risk scores from dashboard "before each" hook for "host risk enable button should redirect to entity management page" "before each" hook for "host risk enable button should redirect to entity management page"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #1 / Entity Analytics Dashboard in Serverless "before each" hook for "should display a splash screen when visited with Security essentials PLI " "before each" hook for "should display a splash screen when visited with Security essentials PLI "
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #3 / Entity Analytics Dashboard new risk score When risk engine is enabled Without data (before the risk engine runs for the first time) "before each" hook for "shows no data detected prompt for host and user risk scores" "before each" hook for "shows no data detected prompt for host and user risk scores"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #3 / Entity Analytics Dashboard new risk score Without data "before each" hook for "shows enable host and user risk button" "before each" hook for "shows enable host and user risk button"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #2 / Entity Analytics Dashboard With anomalies data "before all" hook for "should enable a job and renders the table with pagination" "before all" hook for "should enable a job and renders the table with pagination"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #1 / Entity analytics management page "before each" hook for "renders page as expected" "before each" hook for "renders page as expected"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #2 / Entity Flyout User details "before each" hook for "should display entity flyout and open risk input panel" "before each" hook for "should display entity flyout and open risk input panel"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #3 / EQL rules Detection rules, EQL "before each" hook for "Creates and enables a new EQL rule" "before each" hook for "Creates and enables a new EQL rule"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Event Rendered View "before each" hook for "Event Summary Column" "before each" hook for "Event Summary Column"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Events Viewer Events behavior "before each" hook for "filters the events by applying filter criteria from the search bar at the top of the page" "before each" hook for "filters the events by applying filter criteria from the search bar at the top of the page"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Events Viewer Events viewer fields behaviour "before each" hook for "adds a field to the events viewer when the user clicks the checkbox" "before each" hook for "adds a field to the events viewer when the user clicks the checkbox"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Events Viewer Fields rendering "before each" hook for "displays "view all" option by default" "before each" hook for "displays "view all" option by default"
[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #1 / Exceptions match_any "before each" hook for "Creates exception item" "before each" hook for "Creates exception item"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / Expandable flyout left panel correlations "before each" hook for "should render correlations details correctly" "before each" hook for "should render correlations details correctly"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Expandable flyout left panel threat intelligence "before each" hook for "should serialize its state to url" "before each" hook for "should serialize its state to url"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Expandable flyout state sync "before each" hook for "should test flyout url sync" "before each" hook for "should test flyout url sync"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #1 / Export rules "before each" hook for "exports a custom rule" "before each" hook for "exports a custom rule"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Export timelines Exports a custom timeline template Exports a custom timeline template
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Fields Browser Editing the timeline "before each" hook for "should add/remove columns from the alerts table when the user checks/un-checks them" "before each" hook for "should add/remove columns from the alerts table when the user checks/un-checks them"
[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #1 / Filter Lists "before each" hook for "Filters exception lists on search" "before each" hook for "Filters exception lists on search"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Histogram legend hover actions "before each" hook for "should should add a filter in to KQL bar" "before each" hook for "should should add a filter in to KQL bar"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #2 / Hover actions "before each" hook for "Adds global filter - filter in" "before each" hook for "Adds global filter - filter in"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #2 / Import rules "before each" hook for "Imports a custom rule with exceptions" "before each" hook for "Imports a custom rule with exceptions"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #3 / indicator match Detection rules, Indicator Match Creating new indicator match rules Index patterns "before each" hook for "Contains a predefined index pattern" "before each" hook for "Contains a predefined index pattern"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Inspect Timeline inspects the timeline inspects the timeline
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #6 / Investigate in timeline From alerts table "before each" hook for "should open new timeline from alerts table" "before each" hook for "should open new timeline from alerts table"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #3 / Machine Learning Detection Rules - Editing without Alert Suppression "before each" hook for "allows editing of a rule to add suppression configuration" "before each" hook for "allows editing of a rule to add suppression configuration"
[job] [logs] Serverless Detection Engine - Exceptions - Security Solution Cypress Tests #2 / Manage lists from "Shared Exception Lists" page Create/Export/Delete List "before each" hook for "Export exception list" "before each" hook for "Export exception list"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Manual rule run "before each" hook for "schedule from rule details page" "before each" hook for "schedule from rule details page"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / New Terms rules Detection rules, New Terms "before each" hook for "Creates and enables a new terms rule" "before each" hook for "Creates and enables a new terms rule"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Open timeline "before each" hook for "should open a timeline via url alone without a saved object id" "before each" hook for "should open a timeline via url alone without a saved object id"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Open timeline modal "before each" hook for "should display timeline info in the open timeline modal" "before each" hook for "should display timeline info in the open timeline modal"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #6 / Opening alert previews from alert details flyout "before each" hook for "should render alert preview" "before each" hook for "should render alert preview"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Overflow items Network stats and tables "before each" hook for "Shows more items in the popover" "before each" hook for "Shows more items in the popover"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #2 / Overview Page "before each" hook for "Host stats render with correct values" "before each" hook for "Host stats render with correct values"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Pagination All users and all Hosts tables "before each" hook for "reset all Hosts pagination when sorting column" "before each" hook for "reset all Hosts pagination when sorting column"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Pagination Host uncommon processes table) "before each" hook for "pagination updates results and page number" "before each" hook for "pagination updates results and page number"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / persistent timeline "before each" hook for "persist the deletion of a column" "before each" hook for "persist the deletion of a column"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Ransomware Detection Alerts Ransomware in Alerts Page "before each" hook for "should show ransomware alerts on alerts page" "before each" hook for "should show ransomware alerts on alerts page"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Ransomware Detection Alerts Ransomware in Timelines "before each" hook for "should show ransomware entries in timelines table" "before each" hook for "should show ransomware entries in timelines table"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / Ransomware Prevention Alerts Ransomware in Alerts Page "before each" hook for "should show ransomware alerts on alerts page" "before each" hook for "should show ransomware alerts on alerts page"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / Ransomware Prevention Alerts Ransomware in Timelines "before each" hook for "should render ransomware entries in timelines table" "before each" hook for "should render ransomware entries in timelines table"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #1 / Related integrations integrations not installed rules management table "before each" hook for "should display a badge with the installed integrations" "before each" hook for "should display a badge with the installed integrations"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #2 / risk tab with new risk score "before each" hook for "renders risk tab" "before each" hook for "renders risk tab"
[job] [logs] Serverless Entity Analytics - Security Cypress Tests #3 / risk tab with new risk score "before each" hook for "renders the table" "before each" hook for "renders the table"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Rule actions PLI essentials product tier "before each" hook for "only 3 rule actions should be available" "before each" hook for "only 3 rule actions should be available"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #5 / Rule deletion "before each" hook for "User can delete an individual rule" "before each" hook for "User can delete an individual rule"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #3 / rule snoozing "before each" hook for "ensures the rule is snoozed on the rules management page, rule details page and rule editing page" "before each" hook for "ensures the rule is snoozed on the rules management page, rule details page and rule editing page"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #3 / Rules override "before each" hook for "Creates and enables a new custom rule with override option" "before each" hook for "Creates and enables a new custom rule with override option"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #5 / Rules table: auto-refresh "before each" hook for "gets deactivated when any rule selected and activated after rules unselected" "before each" hook for "gets deactivated when any rule selected and activated after rules unselected"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #1 / Rules table: filtering Last response filter "before each" hook for "Filters rules by last response" "before each" hook for "Filters rules by last response"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #2 / Rules table: links "before each" hook for "should render correct link for rule name - rules" "before each" hook for "should render correct link for rule name - rules"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #3 / Rules table: persistent state while on a happy path "before each" hook for "activates management tab by default" "before each" hook for "activates management tab by default"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #4 / Rules table: selection "before each" hook for "should correctly update the selection label when rules are individually selected and unselected" "before each" hook for "should correctly update the selection label when rules are individually selected and unselected"
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #5 / Rules table: sorting "before each" hook for "Sorts by enabled rules" "before each" hook for "Sorts by enabled rules"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #2 / Saved query rules Custom saved_query detection rule creation "before each" hook for "Creates saved query rule" "before each" hook for "Creates saved query rule"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #2 / SERVERLESS - pinned filters "before each" hook for "show pinned filters on security" "before each" hook for "show pinned filters on security"
[job] [logs] FTR Configs #15 / Serverless Index Management APIs Create enrich policy Can retrieve matching indices
[job] [logs] FTR Configs #34 / Serverless Index Management APIs Create enrich policy Can retrieve matching indices
[job] [logs] FTR Configs #106 / Serverless Index Management APIs Create enrich policy Can retrieve matching indices
[job] [logs] FTR Configs #15 / Serverless Index Management APIs Create enrich policy Can retrieve matching indices
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #1 / Sourcerer "before each" hook for "correctly loads SIEM data view" "before each" hook for "correctly loads SIEM data view"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #4 / Threshold rules "before each" hook for "Creates and enables a new threshold rule" "before each" hook for "Creates and enables a new threshold rule"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #1 / Timeline notes tab "before each" hook for "should render mockdown" "before each" hook for "should render mockdown"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Timeline Pagination "before each" hook for "should paginate records correctly" "before each" hook for "should paginate records correctly"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Timeline query tab "before each" hook for "should display the right query and filters" "before each" hook for "should display the right query and filters"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Timeline search and filters timeline search or filter KQL bar "before each" hook for "should execute a KQL query" "before each" hook for "should execute a KQL query"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Timeline search and filters Update kqlMode for timeline "before each" hook for "should be able to update timeline kqlMode with filter" "before each" hook for "should be able to update timeline kqlMode with filter"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #6 / Timeline table Row Actions "before each" hook for "should refresh the table when alert status is changed" "before each" hook for "should refresh the table when alert status is changed"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #6 / Toggle full screen "before each" hook for "Should hide timeline header and tab list area" "before each" hook for "Should hide timeline header and tab list area"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Unified Timeline table Row Actions "before each" hook for "should refresh the table when alert status is changed" "before each" hook for "should refresh the table when alert status is changed"
[job] [logs] Serverless Investigations - Security Solution Cypress Tests #6 / Unsaved Timeline query tab "before each" hook for "should be able inspect without any issues" "before each" hook for "should be able inspect without any issues"
[job] [logs] Serverless Explore - Security Solution Cypress Tests #1 / Users stats and tables Users page tabs "before each" hook for "renders all users" "before each" hook for "renders all users"
[job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #3 / value lists management modal "before each" hook for "can open and close the modal" "before each" hook for "can open and close the modal"

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…services (#189613) ## Summary While working on #188737 I had to move `supertestWithoutAuth` into `kbn-ftr-common-functional-services` package. This change seems to be bigger than initially planned. Moving it to the separate PR with following changes: - move FTR `SupertestWithoutAuthProvider` service to package - remove "duplicates" in favour of service from package - update service type where needed

dmlemeshko · 2024-08-06T07:22:42Z

Work is finished in #189853

## Summary ### This PR introduces a new type of API integration tests in FTR: deployment-agnostic ![8zcgq0 (1)](https://github.com/user-attachments/assets/17c6d4ee-7848-4a4c-a006-7ae54e523243) #### Test suite is considered deployment-agnostic when it fulfils the following criteria: **Functionality**: It tests Kibana APIs that are **logically identical in both stateful and serverless environments** for the same SAML roles. **Design**: The test design is **clean and does not require additional logic** to execute in either stateful or serverless environments. ### How It Works Most existing stateful tests use basic authentication for API testing. In contrast, serverless tests use SAML authentication with project-specific role mapping. Since stateful deployments also support SAML, deployment-agnostic tests **configure Elasticsearch and Kibana with SAML authentication in both cases**. For roles, stateful deployments define 'viewer', 'editor', and 'admin' roles with serverless-alike privileges. New `samlAuth` service has `AuthProvider` interface with 2 different implementations: depending on environment context (serverless or stateful) appropriate implementation is used. But it remains on service level and hidden in test suite. test example ``` export default function ({ getService }: DeploymentAgnosticFtrProviderContext) { const samlAuth = getService('samlAuth'); const supertestWithoutAuth = getService('supertestWithoutAuth'); let roleAuthc: RoleCredentials; let internalHeaders: InternalRequestHeader; describe('GET /api/console/api_server', () => { before(async () => { roleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('admin'); internalHeaders = samlAuth.getInternalRequestHeader(); }); after(async () => { await samlAuth.invalidateM2mApiKeyWithRoleScope(roleAuthc); }); it('returns autocomplete definitions', async () => { const { body } = await supertestWithoutAuth .get('/api/console/api_server') .set(roleAuthc.apiKeyHeader) .set(internalHeaders) .set('kbn-xsrf', 'true') .expect(200); expect(body.es).to.be.ok(); const { es: { name, globals, endpoints }, } = body; expect(name).to.be.ok(); expect(Object.keys(globals).length).to.be.above(0); expect(Object.keys(endpoints).length).to.be.above(0); }); }); } ``` Please read [readme](https://github.com/elastic/kibana/blob/966822ac872c71284258faf61682176251bcf2c2/x-pack/test/api_integration/deployment_agnostic/README.md) for more details and step-by-step guide. It should help migrating existing serverless tests to deployment-agnostic, assuming requirements are met. ### Examples Deployment-agnostic tests: ``` x-pack/test/api_integration/deployment_agnostic/apis/console/spec_definitions.ts x-pack/test/api_integration/deployment_agnostic/apis/core/compression.ts x-pack/test/api_integration/deployment_agnostic/apis/painless_lab/painless_lab.ts ``` Configs to run it: ``` node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/oblt.serverless.config.ts node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/search.serverless.config.ts node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/security.serverless.config.ts node scripts/functional_tests --config x-pack/test/api_integration/deployment_agnostic/stateful.config.ts ``` PR is a compact version of #188737 with reduced changes in existing serverless tests. --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: elena-shostak <[email protected]> Co-authored-by: Aleh Zasypkin <[email protected]>

dmlemeshko added 3 commits July 19, 2024 13:07

[kbn-ftr-common-functional-services] add saml_auth_provider service

8b438df

Merge remote-tracking branch 'upstream/main' into ftr/unify-saml-auth…

cfe0f90

…-service

update SessionManager constructor

7d8c780

dmlemeshko commented Jul 22, 2024

View reviewed changes

dmlemeshko added 2 commits July 22, 2024 11:30

rename provider

dd7c78c

return common & internal headers from saml_auth service

002efd1

kibanamachine and others added 2 commits July 22, 2024 10:20

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

741e645

…fix'

Merge branch 'main' into ftr/unify-saml-auth-service

1bbf300

dmlemeshko requested a review from mgiota July 22, 2024 15:56

dmlemeshko added 2 commits July 25, 2024 16:12

Merge remote-tracking branch 'upstream/main' into ftr/unify-saml-auth…

b889c3b

…-service

fix InternalRequestHeader import

80e1211

jeramysoucy mentioned this pull request Jul 25, 2024

Leverage deployment agnostic SamlAuthProvider for stateful testing #189222

Open

fix typing

b2fd5cc

defined response type

dcfd81d

dmlemeshko added 3 commits July 26, 2024 16:56

fix typing error in tests

1cd8653

Merge remote-tracking branch 'upstream/main' into ftr/unify-saml-auth…

7ade4d4

…-service

deployment agnostic context provider, base configs, test example

0df6c59

dmlemeshko added 4 commits July 28, 2024 11:06

list default configs in manifests

468da74

Merge remote-tracking branch 'upstream/main' into ftr/unify-saml-auth…

f67c474

…-service

update error message for auth helper

754f0e2

remove comment

c3f7a7d

dmlemeshko added 2 commits July 29, 2024 10:20

remove callback to keep Response type defined

7227e37

Merge branch 'main' into ftr/unify-saml-auth-service

284a71d

dmlemeshko added 3 commits August 2, 2024 13:00

Merge branch 'main' into ftr/unify-saml-auth-service

a31b9b6

Revert "revert test change"

cd4d801

This reverts commit 22fedf1.

Merge branch 'ftr/unify-saml-auth-service' of github.com:dmlemeshko/k…

de35590

…ibana into ftr/unify-saml-auth-service

kibanamachine and others added 2 commits August 2, 2024 11:14

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

fae3087

…fix'

Merge branch 'main' into ftr/unify-saml-auth-service

698ca6a

mgiota reviewed Aug 2, 2024

View reviewed changes

dmlemeshko added 2 commits August 2, 2024 17:11

fix typo, enable configs for ci

684b978

Merge branch 'main' into ftr/unify-saml-auth-service

0475406

dmlemeshko added 2 commits August 2, 2024 17:46

leave samlAuth in common services, but avoid loading by default

6a2a6f4

Merge branch 'ftr/unify-saml-auth-service' of github.com:dmlemeshko/k…

a1b2726

…ibana into ftr/unify-saml-auth-service

dmlemeshko changed the title ~~[FTR] add deployment agnostic SamlAuthProvider service~~ [FTR] support "deployment agnostic" api-integration tests Aug 2, 2024

dmlemeshko added 3 commits August 2, 2024 18:42

remove test plugins from default config

3777dd9

update tests and fix service loading for security solution tests

40bf2c4

Merge branch 'main' into ftr/unify-saml-auth-service

bad93a7

kibanamachine and others added 4 commits August 2, 2024 18:23

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

33a0403

…fix'

update service to fetch internal header

c4b2133

Merge branch 'ftr/unify-saml-auth-service' of github.com:dmlemeshko/k…

b66f587

…ibana into ftr/unify-saml-auth-service

Merge branch 'main' into ftr/unify-saml-auth-service

d489129

dmlemeshko mentioned this pull request Aug 4, 2024

[FTR] support "deployment agnostic" api-integration tests #189853

Merged

dmlemeshko closed this Aug 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FTR] support "deployment agnostic" api-integration tests #188737

[FTR] support "deployment agnostic" api-integration tests #188737

dmlemeshko commented Jul 19, 2024 •

edited

Loading

dmlemeshko Jul 22, 2024

dmlemeshko Jul 22, 2024 •

edited

Loading

dmlemeshko commented Jul 22, 2024

dmlemeshko commented Jul 22, 2024

dmlemeshko commented Jul 25, 2024

dmlemeshko commented Jul 26, 2024

dmlemeshko commented Jul 26, 2024

dmlemeshko commented Jul 26, 2024

dmlemeshko commented Jul 28, 2024

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

mgiota Aug 2, 2024

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

kibana-ci commented Aug 2, 2024 •

edited

Loading

dmlemeshko commented Aug 6, 2024 •

edited

Loading

[FTR] support "deployment agnostic" api-integration tests #188737

[FTR] support "deployment agnostic" api-integration tests #188737

Conversation

dmlemeshko commented Jul 19, 2024 • edited Loading

Summary

dmlemeshko Jul 22, 2024

Choose a reason for hiding this comment

dmlemeshko Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

dmlemeshko commented Jul 22, 2024

dmlemeshko commented Jul 22, 2024

dmlemeshko commented Jul 25, 2024

dmlemeshko commented Jul 26, 2024

dmlemeshko commented Jul 26, 2024

dmlemeshko commented Jul 26, 2024

dmlemeshko commented Jul 28, 2024

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

mgiota Aug 2, 2024

Choose a reason for hiding this comment

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

dmlemeshko commented Aug 2, 2024

kibana-ci commented Aug 2, 2024 • edited Loading

⏳ Build in-progress, with failures

Failed CI Steps

Test Failures

History

dmlemeshko commented Aug 6, 2024 • edited Loading

dmlemeshko commented Jul 19, 2024 •

edited

Loading

dmlemeshko Jul 22, 2024 •

edited

Loading

kibana-ci commented Aug 2, 2024 •

edited

Loading

dmlemeshko commented Aug 6, 2024 •

edited

Loading