[ML] Store ml job configurations in the new .ml-config index #36698

davidkyle · 2018-12-17T10:23:21Z

Store ml anomaly detector jobs and datafeed configurations in the .ml-config index.
Prior to this change job configs were stored in the clusterstate, with this change all new jobs will stored in the new .ml-config index. Version 7 will expect all ml configurations to be stored in the index rather than clusterstate as a step towards that goal all closed job configs will be migrated from the clusterstate to the config index after upgrading.

This commit merges work undertaken on a feature branch.
See #32905

Index mappings for the configuration documents

…3273)

#33994)

This changes the delete filter action to search for jobs using the filter to be deleted in the index rather than the cluster state.

Enables the ml integration tests excluding the rolling upgrade tests and a lot of fixes to make the tests pass again.

Conflicts due to the deprecation of AbstractComponent and removal of settings

This is the 6.6/6.7 implementation of a master node service to keep track of the native process memory requirement of each ML job with an associated native process. The new ML memory tracker service works when the whole cluster is upgraded to at least version 6.6. For mixed version clusters the old mechanism of established model memory stored on the job in cluster state is used. This means that the old (and complex) code to keep established model memory up to date on the job object cannot yet be removed. When this change is forward ported to 7.0 the old way of keeping established model memory updated will be removed.

This enables calls to the job and datafeed APIs in a mixed cluster state before jobs have been migrated

Because the cluster was expanded from 1 node to 3 indices would initially start off with 0 replicas. If the original node was killed before auto-expansion to 1 replica was complete then the test would fail because the indices would be unavailable.

Job updates can apply to cluster state or index jobs this includes reverting a model snapshot and finalizing the job

)

#35598) Small fixes to read from all locations and added index mappings for DelayedDataCheckConfig

After #36069 the approach for reallocating ML persistent tasks after refreshing job memory requirements can be simplified.

Two problems: 1. Stack overflow during async iteration when lots of jobs on same machine 2. Not effectively setting search size in all cases

The same cases are covered in MlMigrationIT

We don't need a Future to wait for completion

elasticmachine · 2018-12-17T10:23:22Z

Pinging @elastic/ml-core

…ion (#36700) This commit adds a cluster settings called `xpack.ml.enable_config_migration`. The setting is `true` by default. When set to `false`, no config migration will be attempted and non-migrated resources (e.g. jobs, datafeeds) will be able to be updated normally. Relates #32905

davidkyle · 2018-12-17T17:23:17Z

retest this please

…6716) Relates #32905

davidkyle · 2018-12-18T15:31:01Z

run gradle build tests 2

droberts195

LGTM on the basis these changes have been previously reviewed in the feature branch PRs

Changes the feature usage retrieval to use the job manager rather than directly talking to the cluster state, because jobs can now be either in cluster state or stored in an index This is a follow-up of #36702 / #36698

davidkyle and others added 30 commits October 18, 2018 10:34

[ML] Job and datafeed mappings with index template (#32719)

0fc87c8

Index mappings for the configuration documents

[ML] Job config document CRUD operations (#32738)

44322b1

[ML] Datafeed config CRUD operations (#32854)

2e6c9ed

[ML] Change JobManager to work with Job config in index (#33064)

e4eb4d7

Fix check style and compilation error after rebase

3b98b3a

[ML] Change Datafeed actions to read config from the config index (#3…

07cceaf

…3273)

[ML] Allocate jobs based on JobParams rather than cluster state config (

3703fbb

#33994)

[ML] Return missing job error when .ml-config is does not exist (#34177)

0803d09

[ML] Close job in index (#34217)

afb238b

[ML] Adjust finalize job action to work with documents (#34226)

47787b3

[ML] Job in index: Datafeed node selector (#34218)

9874b2f

[ML] Job in Index: Stop and preview datafeed (#34605)

7c8b98e

[ML] Delete job document (#34595)

2794d12

[ML] Convert job data remover to work with index configs (#34532)

33cf46e

[ML] Job in index: delete filter action (#34642)

6d36bb8

This changes the delete filter action to search for jobs using the filter to be deleted in the index rather than the cluster state.

[ML] Job in index: Get datafeed and job stats from index (#34645)

a646f8e

[ML] Job in Index: Convert get calendar events to index docs (#34710)

1cac6ba

Merge branch '6.x' into feature-jindex-6x

344b2ab

[ML] Job in Index: Enable integ tests (#34851)

625e675

Enables the ml integration tests excluding the rolling upgrade tests and a lot of fixes to make the tests pass again.

Merge branch '6.x' into feature-jindex-6x

61b178a

Merge branch '6.x' into feature-jindex-6x

3a0a5e7

Conflicts due to the deprecation of AbstractComponent and removal of settings

[ML] Job In Index: Enable GET APIS in mixed state (#35344)

64ba62a

This enables calls to the job and datafeed APIs in a mixed cluster state before jobs have been migrated

[ML] Job in index: Restore ability to update cluster state jobs (#35539)

0046d55

Job updates can apply to cluster state or index jobs this includes reverting a model snapshot and finalizing the job

Merge branch '6.x' into feature-jindex-6x

89ca6f9

Merge branch '6.x' into feature-jindex-6x

971681e

[ML] Job in index: Enable delete actions for clusterstate config (#35590

eb6b75e

)

Merge branch '6.x' into feature-jindex-6x

49781b9

[ML] Job in index: Enable get and update actions for clusterstate jobs (

15a9fdf

#35598) Small fixes to read from all locations and added index mappings for DelayedDataCheckConfig

droberts195 and others added 6 commits December 14, 2018 13:24

[ML] Adapt to periodic persistent task refresh (#36494)

e0a58c9

After #36069 the approach for reallocating ML persistent tasks after refreshing job memory requirements can be simplified.

Merge branch '6.x' into feature-jindex-6x

a2c53ab

Fix TooManyJobsIT.testMultipleNodes

8ad6379

Two problems: 1. Stack overflow during async iteration when lots of jobs on same machine 2. Not effectively setting search size in all cases

[ML] Remove unreliable test assertions

2c3e70a

The same cases are covered in MlMigrationIT

Use execute() instead of submit() in MlMemoryTracker

ed26ae8

We don't need a Future to wait for completion

[ML] JIindex: Limit the size of bulk migrations (#36481)

28b99f0

davidkyle added the :ml Machine learning label Dec 17, 2018

davidkyle and others added 3 commits December 17, 2018 11:53

Merge branch '6.x' into feature-jindex-6x

5d01a32

[ML] Snapshot ml configs before migrating (#36645)

ec4601e

dimitris-athanasiou and others added 2 commits December 17, 2018 22:36

[FEATURE][ML] Split in batches and migrate all jobs and datafeeds (#3…

b6b1f0d

…6716) Relates #32905

Merge branch '6.x' into feature-jindex-6x

fb7e3f5

droberts195 approved these changes Dec 18, 2018

View reviewed changes

davidkyle merged commit 66a582a into 6.x Dec 18, 2018

tvernum mentioned this pull request Dec 19, 2018

[CI] Failures on 6.x for 60_ml_config_migration/Test old cluster jobs and datafeeds and delete them #36810

Closed

hendrikmuhs mentioned this pull request Dec 21, 2018

[ML] fix x-pack usage regression caused by index migration #36936

Merged

droberts195 deleted the feature-jindex-6x branch January 8, 2019 09:22

davidkyle mentioned this pull request Jan 18, 2019

[ML] Create the ml-config index #36792

Merged

lcawl mentioned this pull request Jan 18, 2019

[DOCS] Adds ML PRs to 6.6 release notes #37615

Merged

davidkyle added the v6.6.0 label Jan 18, 2019

davidkyle changed the title ~~[ML] Merge the Jindex 6x feature branch~~ [ML] Store ml job configurations in the new .ml-config index Jan 18, 2019

lcawl added the >feature label Jan 18, 2019

codebrain mentioned this pull request Jan 25, 2019

[meta] 6.6.0 Release elastic/elasticsearch-net#3552

Closed

48 tasks

codebrain mentioned this pull request Mar 19, 2019

[meta] 6.7.0 Release elastic/elasticsearch-net#3615

Closed

24 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Store ml job configurations in the new .ml-config index #36698

[ML] Store ml job configurations in the new .ml-config index #36698

davidkyle commented Dec 17, 2018 •

edited

Loading

elasticmachine commented Dec 17, 2018

davidkyle commented Dec 17, 2018

davidkyle commented Dec 18, 2018

droberts195 left a comment

[ML] Store ml job configurations in the new .ml-config index #36698

[ML] Store ml job configurations in the new .ml-config index #36698

Conversation

davidkyle commented Dec 17, 2018 • edited Loading

elasticmachine commented Dec 17, 2018

davidkyle commented Dec 17, 2018

davidkyle commented Dec 18, 2018

droberts195 left a comment

Choose a reason for hiding this comment

davidkyle commented Dec 17, 2018 •

edited

Loading