-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Debug around ML rule execution #189307
Debug around ML rule execution #189307
Commits on Jul 26, 2024
-
Debug around ML rule execution
Let's see if we can't find why these rules aren't generating alerts.
Configuration menu - View commit details
-
Copy full SHA for 4c0b463 - Browse repository at this point
Copy the full SHA 4c0b463View commit details -
None of these are showing up in the build. It's not yet clear whether this is a log level / file descriptor issue, or whether our code just isn't being executed.
Configuration menu - View commit details
-
Copy full SHA for 5ff34e8 - Browse repository at this point
Copy the full SHA 5ff34e8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5924a05 - Browse repository at this point
Copy the full SHA 5924a05View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8c6df4e - Browse repository at this point
Copy the full SHA 8c6df4eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 13d0256 - Browse repository at this point
Copy the full SHA 13d0256View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7bf8523 - Browse repository at this point
Copy the full SHA 7bf8523View commit details -
Debug our rule statuses as we wait for success
Understanding what happens during the first rule execution (if there is one) might help us to understand why we're not generating alerts that first time.
Configuration menu - View commit details
-
Copy full SHA for a30c4f9 - Browse repository at this point
Copy the full SHA a30c4f9View commit details
Commits on Jul 27, 2024
-
If the rule is going to eventually succeed, we should see this eventually resolve if the problem lies in the rule executor's handling of the failure.
Configuration menu - View commit details
-
Copy full SHA for 2027fdb - Browse repository at this point
Copy the full SHA 2027fdbView commit details -
Reduce testing rule interval to 30s
This should give better granularity on the following: * How long it takes for the ML job to become "started" * How long it takes for the metrics to become available
Configuration menu - View commit details
-
Copy full SHA for 87b6e5f - Browse repository at this point
Copy the full SHA 87b6e5fView commit details -
Use the minimum interval of 1m for our test ML rules
See previous commit for context.
Configuration menu - View commit details
-
Copy full SHA for 4d075dc - Browse repository at this point
Copy the full SHA 4d075dcView commit details -
Wait for the ML API to report our job as "ready" in FTR
There's still a chance that the datafeed/job will _no longer_ be ready by the time we hit the failing MKI tests (or maybe the timing issue pops up years from now 😉), but if this makes our tests more consistent we can start to focus on this: better ML integration.
Configuration menu - View commit details
-
Copy full SHA for 3440144 - Browse repository at this point
Copy the full SHA 3440144View commit details
Commits on Jul 29, 2024
-
Log around our ML waiting utilities
Let's see how long these are pausing; that might indicate an issue.
Configuration menu - View commit details
-
Copy full SHA for b4159d5 - Browse repository at this point
Copy the full SHA b4159d5View commit details -
Wait for anomalies before beginning our test
Despite our job being started, we're now receiving _no_ alerts when before we had some. I think this is because the job is starting, but no anomalies are ready yet. This should validate that hypothesis.
Configuration menu - View commit details
-
Copy full SHA for 4853b10 - Browse repository at this point
Copy the full SHA 4853b10View commit details -
Wait for anomalies to be searchable
This is less restrictive than the ML helper, which seems to wait for the job to report as having processed records. Let's see if this implementation works for us.
Configuration menu - View commit details
-
Copy full SHA for 02a5ad1 - Browse repository at this point
Copy the full SHA 02a5ad1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2af7fb5 - Browse repository at this point
Copy the full SHA 2af7fb5View commit details