[Rule Tuning] Updating Rules to Reflect V3 Unsupervised ML Jobs

[dishadasgupta]

Related Issues/PRs

https://github.com/elastic/security-team/issues/1490
elastic/kibana#123274

Summary

The job ids in the rules need to also point to new V3 versions of the jobs, i.e:

machine_learning_job_id = ["windows_rare_metadata_process", "v2_windows_rare_metadata_process", "v3_windows_rare_metadata_process"]

26 Updated Rules:

Linux (14):

1. ml_linux_anomalous_compiler_activity.toml
2. ml_linux_anomalous_metadata_process.toml
3. ml_linux_anomalous_metadata_user.toml
4. ml_linux_anomalous_network_activity.toml
5. ml_linux_anomalous_network_port_activity.toml
6. ml_linux_anomalous_process_all_hosts.toml
7. ml_linux_anomalous_sudo_activity.toml
8. ml_linux_anomalous_user_name.toml
9. ml_linux_system_information_discovery.toml
10. ml_linux_system_network_configuration_discovery.toml
11. ml_linux_system_network_connection_discovery.toml
12. ml_linux_system_process_discovery.toml
13. ml_linux_system_user_discovery.toml
14. ml_rare_process_by_host_linux.toml

Windows (12):

1. ml_rare_process_by_host_windows.toml
2. ml_windows_anomalous_metadata_process.toml
3. ml_windows_anomalous_metadata_user.toml
4. ml_windows_anomalous_network_activity.toml
5. ml_windows_anomalous_path_activity.toml
6. ml_windows_anomalous_process_all_hosts.toml
7. ml_windows_anomalous_process_creation.toml
8. ml_windows_anomalous_script.toml
9. ml_windows_anomalous_service.toml
10. ml_windows_anomalous_user_name.toml
11. ml_windows_rare_user_runas_event.toml
12. ml_windows_rare_user_type10_remote_login.toml

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[w0rk3r]

@dishadasgupta, can you bump the update_date field on these rules?

[randomuserid]

@spong so we're adding a second, or third, job name to the ML rules. In 8.1, we have refactored all of the shipping host based ML jobs - save for 3 that do not work across data types - into two modules, Security: Linux v3 and Security: Windows v3. The advantages of shipping a third module are these;

• one module to rule them all. Users with relatively recent data - from the past year or so - should never need to read docs and try to figure out where to use the v1 module jobs and the v2 module jobs. They can just use the v3 module
• cleaner migration - no name conflicts or other errors that would result from placing the new jobs in the v2 module. Users can cleanly migrate by stopping the v2 jobs and starting the v3 jobs
• reduced risk of overwriting of jobs and loss of models and / or data with this simpler migration path

[spong]

Thanks for the ping @randomuserid! This is fantastic news! 🙌 🎉 🙂

Thinking through this, we may have one more issue to work through if we don't plan on shipping new rules as part of these new modules.

When @rylnd updated ML Rules to support multiple jobs (elastic/kibana#97073 (review)), we had to decide on the different partial failure scenarios when a job isn't running or its module isn't installed. In this instance, I believe if you only have the v3 module installed/running and the other two aren't installed, the rule may sit in a failure state and not write alerts:

3. All but one job is healthy, and unhealthy job is in a module that has not been installed === rule failure, no alerts written

When developing the new module you didn't happen to test this by duplicating one the existing rules and just adding your v3 job manually did you? If this is indeed the case, we'll either need to ship new pre-packaged ML Rules, or add some special logic for this. Going the new rule route would add confusion if we can't deprecate the old ones, and it'd be preferred to not add special logic to rules for pre-built rules.

[brokensound77]

Changes LGTM, but IINM, I believe the rules will fail if the job_id does not exist. I am not totally sure if that is the same behavior when it is an array and if it expects all job_ids to exist. Maybe you can make a test rule to verify?

If so, we would need to add/bump min_stack_version to match the stack version in which the jobs were added

[randomuserid]

@dishadasgupta I think we can close this for now b/c I'm not sure when we will be ready to merge. I will first need to check on issues with the ML rules and the question of modules.

[brokensound77]

@randomuserid @dishadasgupta can this PR be closed? Last update I saw was that a new PR would be opened with the revised approach

[randomuserid]

@randomuserid @dishadasgupta can this PR be closed? Last update I saw was that a new PR would be opened with the revised approach

@brokensound77 yes I will close this one & circle back with you and @spong on how to compose the rules after we consolidate down to a single module for Linux and Windows jobs.