Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor a11y tests to group by data sets #113876

Merged
merged 22 commits into from
Oct 6, 2021
Merged

Conversation

LeeDr
Copy link
Contributor

@LeeDr LeeDr commented Oct 5, 2021

Summary

Most of the a11y tests use sample flights data set. But most of them navigated to '/tutorial_directory/sampleData' and at least checked that the data was installed. And only one of them was removing the data. So I created an index file and moved the sample data loading to a before method there. 3 tests either use no data, or load different data and they're in a separate describe block.

Checklist

Delete any items that are not applicable to this PR.

@LeeDr LeeDr requested a review from bhavyarm October 5, 2021 00:38
@LeeDr LeeDr added auto-backport Deprecated - use backport:version if exact versions are needed Project:Accessibility v7.16.0 v8.0.0 v7.15.1 labels Oct 5, 2021
@bhavyarm
Copy link
Contributor

bhavyarm commented Oct 5, 2021

@elasticmachine merge upstream

@LeeDr
Copy link
Contributor Author

LeeDr commented Oct 6, 2021

@elasticmachine merge upstream

Unrelated Jest test failure:

console.log
usageCollection.reportUiCounter is unavailable. Ensure this is setup via .

  at x-pack/plugins/observability/public/hooks/use_track_metric.tsx:61:15

@LeeDr
Copy link
Contributor Author

LeeDr commented Oct 6, 2021

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/uptime/certificates·ts.Uptime app with generated data certificates with certs page "before each" hook for "displays certificates"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://github.com/elastic/kibana/issues/112900

[00:00:00]     │
[00:42:39]       └-: Uptime app
[00:42:39]         └-> "before all" hook in "Uptime app"
[00:42:39]         └-: with generated data
[00:42:39]           └-> "before all" hook in "with generated data"
[00:43:29]           └-: certificates
[00:43:29]             └-> "before all" hook in "certificates"
[00:43:43]             └-: with certs
[00:43:43]               └-> "before all" hook for "can navigate to cert page"
[00:43:43]               └-> "before all" hook for "can navigate to cert page"
[00:43:43]                 │ info [x-pack/test/functional/es_archives/uptime/blank] Loading "mappings.json"
[00:43:43]                 │ info [x-pack/test/functional/es_archives/uptime/blank] Loading "data.json"
[00:43:43]                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [heartbeat-8-generated-test] creating index, cause [api], templates [], shards [1]/[1]
[00:43:43]                 │ info [x-pack/test/functional/es_archives/uptime/blank] Created index "heartbeat-8-generated-test"
[00:43:43]                 │ debg [x-pack/test/functional/es_archives/uptime/blank] "heartbeat-8-generated-test" settings undefined
[00:43:43]                 │ debg TestSubjects.exists(uptimeSettingsToOverviewLink)
[00:43:43]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="uptimeSettingsToOverviewLink"]') with timeout=0
[00:43:43]                 │ debg --- retry.tryForTime error: [data-test-subj="uptimeSettingsToOverviewLink"] is not displayed
[00:43:44]                 │ debg navigating to uptime url: http://localhost:61181/app/uptime
[00:43:44]                 │ debg navigate to: http://localhost:61181/app/uptime
[00:43:44]                 │ debg browser[INFO] http://localhost:61181/app/uptime?_t=1633530259143 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:43:44]                 │
[00:43:44]                 │ debg browser[INFO] http://localhost:61181/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:43:44]                 │ debg ... sleep(700) start
[00:43:45]                 │ debg ... sleep(700) end
[00:43:45]                 │ debg returned from get, calling refresh
[00:43:45]                 │ debg browser[INFO] http://localhost:61181/app/uptime?_t=1633530259143 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:43:45]                 │
[00:43:45]                 │ debg browser[INFO] http://localhost:61181/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:43:46]                 │ debg currentUrl = http://localhost:61181/app/uptime
[00:43:46]                 │          appUrl = http://localhost:61181/app/uptime
[00:43:46]                 │ debg TestSubjects.find(kibanaChrome)
[00:43:46]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:43:46]                 │ debg ... sleep(501) start
[00:43:46]                 │ERROR browser[SEVERE] http://localhost:61181/46844/bundles/plugin/uptime/8.0.0/uptime.chunk.1.js 2:3687 "API /api/uptime/monitor/list is not returning expected response, Invalid value \"undefined\" supplied to \"summaries.state.summaryPings.monitor.timespan.lt\" for response" Object
[00:43:46]                 │ debg ... sleep(501) end
[00:43:46]                 │ debg in navigateTo url = http://localhost:61181/app/uptime
[00:43:46]                 │ debg isGlobalLoadingIndicatorVisible
[00:43:46]                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:43:46]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:43:48]                 │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:43:48]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:43:48]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:43:48]                 │ debg TestSubjects.exists(uptimeOverviewPage)
[00:43:48]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="uptimeOverviewPage"]') with timeout=2000
[00:43:49]                 │ debg TestSubjects.click(superDatePickerApplyTimeButton)
[00:43:49]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:43:49]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:43:49]               └-> can navigate to cert page
[00:43:49]                 └-> "before each" hook: global before each for "can navigate to cert page"
[00:43:49]                 └-> "before each" hook: delete settings for "can navigate to cert page"
[00:43:49]                   │ debg Deleting saved object {
[00:43:49]                   │        type: 'uptime-dynamic-settings',
[00:43:49]                   │        id: 'uptime-dynamic-settings-singleton'
[00:43:49]                   │      }/%s
[00:43:49]                 └-> "before each" hook: load heartbeat data for "can navigate to cert page"
[00:43:49]                   │ info [x-pack/test/functional/es_archives/uptime/blank] Loading "mappings.json"
[00:43:49]                   │ info [x-pack/test/functional/es_archives/uptime/blank] Loading "data.json"
[00:43:49]                   │ info [o.e.c.m.MetadataDeleteIndexService] [node-01] [heartbeat-8-generated-test/XdTdj8wfRpGjdNpnc5JgYw] deleting index
[00:43:49]                   │ info [x-pack/test/functional/es_archives/uptime/blank] Deleted existing index "heartbeat-8-generated-test"
[00:43:49]                   │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [heartbeat-8-generated-test] creating index, cause [api], templates [], shards [1]/[1]
[00:43:49]                   │ERROR browser[SEVERE] http://localhost:61181/46844/bundles/plugin/uptime/8.0.0/uptime.chunk.1.js 2:3687 "API /api/uptime/monitor/list is not returning expected response, Invalid value \"undefined\" supplied to \"summaries.state.summaryPings.monitor.timespan.lt\" for response" Object
[00:43:49]                   │ info [x-pack/test/functional/es_archives/uptime/blank] Created index "heartbeat-8-generated-test"
[00:43:49]                   │ debg [x-pack/test/functional/es_archives/uptime/blank] "heartbeat-8-generated-test" settings undefined
[00:43:49]                 └-> "before each" hook for "can navigate to cert page"
[00:43:49]                 │ debg isGlobalLoadingIndicatorVisible
[00:43:49]                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:43:49]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:43:51]                 │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:43:51]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:43:51]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:43:51]                 │ debg TestSubjects.exists(data-missing)
[00:43:51]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="data-missing"]') with timeout=2500
[00:43:54]                 │ debg --- retry.tryForTime error: [data-test-subj="data-missing"] is not displayed
[00:43:54]                 │ debg TestSubjects.missingOrFail(data-missing)
[00:43:54]                 │ debg Find.waitForDeletedByCssSelector('[data-test-subj="data-missing"]') with timeout=2500
[00:43:55]                 │ debg Find.existsByCssSelector('[href="/app/uptime/certificates"]') with timeout=2500
[00:43:55]                 │ debg TestSubjects.exists(uptimeCertificatesPage)
[00:43:55]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="uptimeCertificatesPage"]') with timeout=0
[00:43:55]                 │ debg --- retry.tryForTime error: [data-test-subj="uptimeCertificatesPage"] is not displayed
[00:43:55]                 │ debg Find.existsByCssSelector('[href="/app/uptime/certificates"]') with timeout=0
[00:43:55]                 │ debg Find.clickByCssSelector('[href="/app/uptime/certificates"]') with timeout=10000
[00:43:55]                 │ debg Find.findByCssSelector('[href="/app/uptime/certificates"]') with timeout=10000
[00:43:55]                 │ debg TestSubjects.exists(uptimeCertificatesPage)
[00:43:55]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="uptimeCertificatesPage"]') with timeout=120000
[00:43:55]                 └- ✓ pass  (6.3s) "Uptime app with generated data certificates with certs can navigate to cert page"
[00:43:55]               └-> "after each" hook: unload for "can navigate to cert page"
[00:43:55]                 │ info [x-pack/test/functional/es_archives/uptime/blank] Unloading indices from "mappings.json"
[00:43:55]                 │ info [o.e.c.m.MetadataDeleteIndexService] [node-01] [heartbeat-8-generated-test/UusVHHnGR9mHH0LLsJiaLA] deleting index
[00:43:55]                 │ info [x-pack/test/functional/es_archives/uptime/blank] Deleted existing index "heartbeat-8-generated-test"
[00:43:55]                 │ info [x-pack/test/functional/es_archives/uptime/blank] Unloading indices from "data.json"
[00:43:55]               └-: page
[00:43:55]                 └-> "before all" hook for "displays certificates"
[00:43:55]                 └-> displays certificates
[00:43:55]                   └-> "before each" hook: global before each for "displays certificates"
[00:43:55]                   └-> "before each" hook: delete settings for "displays certificates"
[00:43:55]                     │ debg Deleting saved object {
[00:43:55]                     │        type: 'uptime-dynamic-settings',
[00:43:55]                     │        id: 'uptime-dynamic-settings-singleton'
[00:43:55]                     │      }/%s
[00:43:56]                   └-> "before each" hook: load heartbeat data for "displays certificates"
[00:43:56]                     │ info [x-pack/test/functional/es_archives/uptime/blank] Loading "mappings.json"
[00:43:56]                     │ info [x-pack/test/functional/es_archives/uptime/blank] Loading "data.json"
[00:43:56]                     │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [heartbeat-8-generated-test] creating index, cause [api], templates [], shards [1]/[1]
[00:43:56]                     │ info [x-pack/test/functional/es_archives/uptime/blank] Created index "heartbeat-8-generated-test"
[00:43:56]                     │ debg [x-pack/test/functional/es_archives/uptime/blank] "heartbeat-8-generated-test" settings undefined
[00:43:56]                   └-> "before each" hook for "displays certificates"
[00:43:56]                   └-> "before each" hook for "displays certificates"
[00:43:56]                     │ debg TestSubjects.exists(uptimeCertificatesPage)
[00:43:56]                     │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="uptimeCertificatesPage"]') with timeout=0
[00:43:56]                     │ debg TestSubjects.click(superDatePickerApplyTimeButton)
[00:43:56]                     │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:43:56]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:44:06]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:44:06]                     │      Wait timed out after 10035ms
[00:44:07]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:44:17]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:44:17]                     │      Wait timed out after 10008ms
[00:44:17]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:44:27]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:44:27]                     │      Wait timed out after 10052ms
[00:44:28]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:44:38]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:44:38]                     │      Wait timed out after 10032ms
[00:44:38]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:44:48]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:44:48]                     │      Wait timed out after 10029ms
[00:44:49]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:44:59]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:44:59]                     │      Wait timed out after 10054ms
[00:44:59]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:45:09]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:45:09]                     │      Wait timed out after 10048ms
[00:45:10]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:45:20]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:45:20]                     │      Wait timed out after 10033ms
[00:45:20]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:45:30]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:45:30]                     │      Wait timed out after 10049ms
[00:45:31]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:45:41]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:45:41]                     │      Wait timed out after 10057ms
[00:45:41]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:45:51]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:45:51]                     │      Wait timed out after 10036ms
[00:45:52]                     │ debg Find.findByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=10000
[00:46:02]                     │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:46:02]                     │      Wait timed out after 10054ms
[00:46:03]                     │ info Taking screenshot "/dev/shm/workspace/parallel/18/kibana/x-pack/test/functional/screenshots/failure/Uptime app with generated data certificates with certs page _before each_ hook for _displays certificates_.png"
[00:46:03]                     │ info Current URL is: http://localhost:61181/app/uptime/certificates
[00:46:03]                     │ info Saving page source to: /dev/shm/workspace/parallel/18/kibana/x-pack/test/functional/failure_debug/html/Uptime app with generated data certificates with certs page _before each_ hook for _displays certificates_.html
[00:46:03]                     └- ✖ fail: Uptime app with generated data certificates with certs page "before each" hook for "displays certificates"
[00:46:03]                     │      Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
[00:46:03]                     │ Wait timed out after 10054ms
[00:46:03]                     │     at /dev/shm/workspace/parallel/18/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:46:03]                     │     at runMicrotasks (<anonymous>)
[00:46:03]                     │     at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:46:03]                     │       at onFailure (/dev/shm/workspace/parallel/18/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:46:03]                     │       at retryForSuccess (/dev/shm/workspace/parallel/18/kibana/test/common/services/retry/retry_for_success.ts:57:13)
[00:46:03]                     │       at RetryService.try (/dev/shm/workspace/parallel/18/kibana/test/common/services/retry/retry.ts:31:12)
[00:46:03]                     │       at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/18/kibana/test/functional/services/common/find.ts:360:5)
[00:46:03]                     │       at TestSubjects.click (/dev/shm/workspace/parallel/18/kibana/test/functional/services/common/test_subjects.ts:105:5)
[00:46:03]                     │       at refreshApp (test/functional/services/uptime/navigation.ts:32:5)
[00:46:03]                     │       at Object.refreshApp (test/functional/services/uptime/navigation.ts:37:7)
[00:46:03]                     │       at Context.<anonymous> (test/functional/apps/uptime/certificates.ts:67:11)
[00:46:03]                     │       at Object.apply (/dev/shm/workspace/parallel/18/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:46:03]                     │ 
[00:46:03]                     │ 

Stack Trace

Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="superDatePickerApplyTimeButton"])
Wait timed out after 10054ms
    at /dev/shm/workspace/parallel/18/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at onFailure (/dev/shm/workspace/parallel/18/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/18/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at RetryService.try (/dev/shm/workspace/parallel/18/kibana/test/common/services/retry/retry.ts:31:12)
    at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/18/kibana/test/functional/services/common/find.ts:360:5)
    at TestSubjects.click (/dev/shm/workspace/parallel/18/kibana/test/functional/services/common/test_subjects.ts:105:5)
    at refreshApp (test/functional/services/uptime/navigation.ts:32:5)
    at Object.refreshApp (test/functional/services/uptime/navigation.ts:37:7)
    at Context.<anonymous> (test/functional/apps/uptime/certificates.ts:67:11)
    at Object.apply (/dev/shm/workspace/parallel/18/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Contributor

@bhavyarm bhavyarm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @LeeDr for doing this!

@bhavyarm bhavyarm marked this pull request as ready for review October 6, 2021 16:02
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-accessibility (Project:Accessibility)

@bhavyarm bhavyarm added the release_note:skip Skip the PR/issue when compiling release notes label Oct 6, 2021
@LeeDr LeeDr merged commit 00de78d into elastic:master Oct 6, 2021
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Oct 6, 2021
* test if 2s sleep fixes flakey test

* switch from sleep to retry.waitFor

* remove .only, add comments to config file about sample data add/remove

* refactor a11y tests for sample data load

* refactor a11y tests for sample data load

* cleanup comments

* cleanup comments

* fix eslint errors

Co-authored-by: Kibana Machine <[email protected]>
@kibanamachine
Copy link
Contributor

💔 Backport failed

Status Branch Result
7.x
7.15 Commit could not be cherrypicked due to conflicts

Successful backport PRs will be merged automatically after passing CI.

To backport manually run:
node scripts/backport --pr 113876

kibanamachine added a commit that referenced this pull request Oct 6, 2021
* test if 2s sleep fixes flakey test

* switch from sleep to retry.waitFor

* remove .only, add comments to config file about sample data add/remove

* refactor a11y tests for sample data load

* refactor a11y tests for sample data load

* cleanup comments

* cleanup comments

* fix eslint errors

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Lee Drengenberg <[email protected]>
@bhavyarm bhavyarm mentioned this pull request Aug 22, 2022
77 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed Project:Accessibility release_note:skip Skip the PR/issue when compiling release notes v7.15.1 v7.16.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants