forked from elastic/kibana
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Hybrid index pattern test (elastic#43498)
* Added 5 second sleep to let thte page load before checking for title. * Made changes to Rollups page object, added data test subjects to some elements, added function to get Indices to indexmanagement page. * Removed 5 second wait from old commit, * Updated home page for a more simplified create rollups method. Simplified data for the rollups to use a simple data structure rather than using makelogs. * Added stop to the previous rollup job that ran. Added some timeouts to wait for the rollup to run. Added future data to be rolled into hybrid index pattern. Added some custom handling for creating different types of index patterns. * Added forking condition for if the create index pattern button has the dropdown due to rollup data being present and to handle that case. Changed assertion for rollup job test to ensure that there is one index created with the proper name from the rollup job. Made adjustment to cron expression to run every 10 seconds. Added deletion clauses for all rollup jobs, index patterns and indices to ensure proper teardown. * Added navigate to the discover page and verify the number of hits. * Fixed assertion: Not supposed to contain hits. * Fixing the button that's being searched for. * Made adjustment to setings page due to test failures. * Made adjustment to setings page due to test failures. * Made a change to the order of the parameters for createIndexPattern which was causing non rollup index patterns to break. Also added new conditional and function to click on rollup index pattern button. * Changed ciGroup and also added logging statements to rollup jobs test. * Trying an additional wait to see if there is a race condition. * Changed timing of when rollup job is stopped. It was getting stopped before having triggered. * Added waitlogic for condition rather than hardcoded skeep. Used cheerio methods to parse table instead of bluebird's map async. Updated tests assertiuons to ensure the rollup has triggered and produced the right number of documents. Tested many times with Throttling and headless mode enabled. * Returned the method reloadIndicesButton back to the page object instead of modifying it. * Changed CI job to run test repeatedly. * Forgot some script omissions. * Reverting the script changes. Did not work. * Readded config to run tests repeatedly per Dima. * Added additional config for running CI tests. * Fixed config error. * Removed extra rollup cration step. * Made edits per review and added some comments to clarify functions purposes. * Added explanation of assertion and change that the rollup_info file was renamed. * Adjusted assertion to compensate for CI's slow network. * Split the hybrid index pattern test out and rollup job to be in separate files. Mocked out rollups and cleaned up log statements. * Added load empty kibana to reset kibana to an empty state. * Removed duplicate file and move method over from it. Updated references. * Renamed test suite to hybrid index pattern as that's what the test entails * Fixed missed reference rename. * Fixed mistaken async keyword causing unhandled rejection error. Also deleted rollup job in hybrid index pattern test in after hook. Renoved log statements. * Reverted changes to build scripts to normal. * Fixed jobs file to match upstream due to a missed fix in a merge conflict. * Added remainder of CI groups to job. * Removed whitespace error. * Made fixes per CJs review. Fixed clashing variable names. Reverted rollup job test to assert using rollups table. Changed group back to ciGroup1. Removed duplicate teardown index deletion calls. * Created more concrete assertion for checking if hybrid index pattern was created. Some refactorings. Changed Math.floor() to Date.now() for avoidance of name collisions. * Removed the data.json file. * Addressed nits.
- Loading branch information
1 parent
2595dd0
commit b56d5c3
Showing
8 changed files
with
293 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
32 changes: 32 additions & 0 deletions
32
x-pack/test/functional/apps/rollup_job/hybrid_index_helper.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License; | ||
| * you may not use this file except in compliance with the Elastic License. | ||
| */ | ||
|
|
||
|
|
||
| export default function mockRolledUpData(jobName, targetIndexName, day) { | ||
| return { | ||
| index: `${targetIndexName}`, | ||
| body: { | ||
| '_rollup.version': 2, | ||
| '@timestamp.date_histogram.time_zone': 'UTC', | ||
| '@timestamp.date_histogram.timestamp': day.toISOString(), | ||
| '@timestamp.date_histogram.interval': '1000ms', | ||
| '@timestamp.date_histogram._count': 1, | ||
| '_rollup.id': jobName | ||
| } | ||
| }; | ||
| } | ||
|
|
||
| //This function just adds some stub indices that includes a timestamp and an arbritary metric. This is fine since we are not actually testing | ||
| //rollup functionality. | ||
| export function mockIndices(day, prepend) { | ||
| return { | ||
| index: `${prepend}-${day.format('MM-DD-YYYY')}`, | ||
| body: { | ||
| '@timestamp': day.toISOString(), | ||
| foo_metric: 1 | ||
| } | ||
| }; | ||
| } |
103 changes: 103 additions & 0 deletions
103
x-pack/test/functional/apps/rollup_job/hybrid_index_pattern.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License; | ||
| * you may not use this file except in compliance with the Elastic License. | ||
| */ | ||
|
|
||
|
|
||
| import datemath from '@elastic/datemath'; | ||
| import expect from '@kbn/expect'; | ||
| import mockRolledUpData, { mockIndices } from './hybrid_index_helper'; | ||
|
|
||
| export default function ({ getService, getPageObjects }) { | ||
|
|
||
| const es = getService('es'); | ||
| const esArchiver = getService('esArchiver'); | ||
| const retry = getService('retry'); | ||
| const PageObjects = getPageObjects(['common', 'settings']); | ||
|
|
||
| describe('hybrid index pattern', function () { | ||
| //Since rollups can only be created once with the same name (even if you delete it), | ||
| //we add the Date.now() to avoid name collision if you run the tests locally back to back. | ||
| const rollupJobName = `hybrid-index-pattern-test-rollup-job-${Date.now()}`; | ||
| const rollupTargetIndexName = `rollup-target-data`; | ||
| const regularIndexPrefix = `regular-index`; | ||
| const rollupSourceIndexPrefix = `rollup-source-data`; | ||
| const rollupIndexPatternName = `${regularIndexPrefix}*,${rollupTargetIndexName}`; | ||
| const now = new Date(); | ||
| const pastDates = [ | ||
| datemath.parse('now-1d', { forceNow: now }), | ||
| datemath.parse('now-2d', { forceNow: now }), | ||
| datemath.parse('now-3d', { forceNow: now }), | ||
| ]; | ||
|
|
||
| it('create hybrid index pattern', async () => { | ||
| //Create data for rollup job to recognize. | ||
| //Index past data to be used in the test. | ||
| await pastDates.map(async (day) => { | ||
| await es.index(mockIndices(day, rollupSourceIndexPrefix)); | ||
| }); | ||
|
|
||
| await retry.waitForWithTimeout('waiting for 3 records to be loaded into elasticsearch.', 10000, async () => { | ||
| const response = await es.indices.get({ | ||
| index: `${rollupSourceIndexPrefix}*`, | ||
| allow_no_indices: false | ||
| }); | ||
| return Object.keys(response).length === 3; | ||
| }); | ||
|
|
||
| await retry.try(async () => { | ||
| //Create a rollup for kibana to recognize | ||
| await es.transport.request({ | ||
| path: `/_rollup/job/${rollupJobName}`, | ||
| method: 'PUT', | ||
| body: { | ||
| 'index_pattern': `${rollupSourceIndexPrefix}*`, | ||
| 'rollup_index': rollupTargetIndexName, | ||
| 'cron': '*/10 * * * * ?', | ||
| 'groups': { | ||
| 'date_histogram': { | ||
| 'fixed_interval': '1000ms', | ||
| 'field': '@timestamp', | ||
| 'time_zone': 'UTC' | ||
| } | ||
| }, | ||
| 'timeout': '20s', | ||
| 'page_size': 1000 | ||
| } | ||
| }); | ||
| }); | ||
|
|
||
|
|
||
| await pastDates.map(async (day) => { | ||
| await es.index(mockRolledUpData(rollupJobName, rollupTargetIndexName, day)); | ||
| }); | ||
|
|
||
| //Index live data to be used in the test. | ||
| await es.index(mockIndices(datemath.parse('now', { forceNow: now }), regularIndexPrefix)); | ||
|
|
||
| await PageObjects.common.navigateToApp('settings'); | ||
| await PageObjects.settings.createIndexPattern(rollupIndexPatternName, '@timestamp', false); | ||
|
|
||
|
|
||
| await PageObjects.settings.clickKibanaIndexPatterns(); | ||
| const indexPattern = (await PageObjects.settings.getIndexPatternList()).pop(); | ||
| const indexPatternText = await indexPattern.getVisibleText(); | ||
| expect(indexPatternText).to.contain(rollupIndexPatternName); | ||
| expect(indexPatternText).to.contain('Rollup'); | ||
| }); | ||
|
|
||
| after(async () => { | ||
| // Delete the rollup job. | ||
| await es.transport.request({ | ||
| path: `/_rollup/job/${rollupJobName}`, | ||
| method: 'DELETE', | ||
| }); | ||
|
|
||
| await es.indices.delete({ index: rollupTargetIndexName }); | ||
| await es.indices.delete({ index: `${regularIndexPrefix}*` }); | ||
| await es.indices.delete({ index: `${rollupSourceIndexPrefix}*` }); | ||
| await esArchiver.load('empty_kibana'); | ||
| }); | ||
| }); | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.