Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs UI] Add dataset filter to ML module setup screen #64470

Merged

Conversation

weltenwort
Copy link
Member

@weltenwort weltenwort commented Apr 24, 2020

Summary

This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen. When no dataset is selected for an index group, no filtering is applied. As soon as one dataset is selected, the datafeed applies a terms filter to the incoming documents.

Note: Even though the datasets are displayed for each index group, the datafeed can't apply the dataset filters selectively when it pushes the data to the ML job. The grouping only serves as an indication of occurrence of the dataset terms.

closes #60390

Previews

image

Testing

  • For performance reasons the dataset terms are fetched for the time interval selected in the preceding setup form section. When that time interval is changed the available datasets should be reloaded.
  • The resulting datafeed should be created with a corresponding filter query in its configuration.

Task breakdown

@weltenwort weltenwort added release_note:enhancement Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.8.0 labels Apr 24, 2020
@weltenwort weltenwort self-assigned this Apr 24, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

@weltenwort weltenwort marked this pull request as ready for review April 30, 2020 23:04
@weltenwort weltenwort requested review from a team as code owners April 30, 2020 23:04
@weltenwort
Copy link
Member Author

@elasticmachine merge upstream

@Kerry350 Kerry350 self-requested a review May 4, 2020 08:08
@weltenwort
Copy link
Member Author

Test failures look unrelated...

@elasticmachine merge upstream

@weltenwort
Copy link
Member Author

@elasticmachine merge upstream

Copy link
Contributor

@Kerry350 Kerry350 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job. Code looks good, and I was able to setup my jobs with "All" as well as a filter for the datasets 👍 One small comment.

startTime: number,
endTime: number
) => {
const response = await npStart.http.fetch(LOG_ANALYSIS_VALIDATE_DATASETS_PATH, {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be changed so as to not use the legacy_singletons, e.g. callValidateDatasetsAPI taking fetch as an argument?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely - I just kept it in sync with the other API calls so far. Would it be ok to switch all of them over at once in a separate PR (that is not subject to a close deadline)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep - of course 👍

Comment on lines -4 to -14
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "message"
}
}
]
}
}
Copy link
Member Author

@weltenwort weltenwort May 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Since the setup API doesn't merge this and any queries supplied at setup time, the query is now only defined via the API.

Copy link
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ML changes LGTM

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@weltenwort weltenwort merged commit 39e31d6 into elastic:master May 4, 2020
@weltenwort weltenwort deleted the logs-ui-ml-setup-add-dataset-filter branch May 4, 2020 11:29
weltenwort added a commit to weltenwort/kibana that referenced this pull request May 4, 2020
This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.
gmmorris added a commit to gmmorris/kibana that referenced this pull request May 4, 2020
…ana into alerting/np-tests-migration

* 'alerting/np-tests-migration' of github.com:gmmorris/kibana:
  [APM] Agent remote config: validation for Java agent configs (elastic#63956)
  [APM] Fix duplicate index patterns (elastic#64883)
  Drilldowns (elastic#61219)
  [Alerting] fix labels and links in PagerDuty action ui and docs (elastic#64032)
  [Event Log] Ensure sorting tests are less flaky (elastic#64781)
  update endpoint to restrict removing with datasources (elastic#64978)
  [Logs UI] [Alerting] Alerts management page enhancements (elastic#64654)
  Adjust kibana app owning files (elastic#65064)
  Migrate tutorial resources (elastic#64298)
  [Logs UI] Tweak copy in log alerts dialog (elastic#64645)
  [Logs UI] [Alerting] Documentation (elastic#64886)
  [Logs UI] Add dataset filter to ML module setup screen (elastic#64470)
  [TSVB] Fixing memory leak (elastic#64918)
  Bump backport to 5.4.1 (elastic#65041)
weltenwort added a commit that referenced this pull request May 4, 2020
…#65063)

Backports the following commits to 7.x:
 - [Logs UI] Add dataset filter to ML module setup screen (#64470)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Logs UI Logs UI feature release_note:enhancement Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.8.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Logs UI] Allow users to specify datasets in ML job configuration
5 participants